acceptodds
Under review as a conference paper at ICLR 2027

Do AI Agents have Normative Agency?

Abstract

AI agents possess significant functional agency to pursue goals, respond to feedback, and act in an autonomous manner. In this work we ask whether they possess normative agency as well: the ability to act for reasons they take themselves to be answerable to, from their own practical standpoint, the position from which they must decide what they ought to do. Existing evaluations largely test whether AI agents recognize, reproduce, or comply with normative reasons, which leaves open whether those reasons have any authority for the agent itself. However, testing for normative agency poses an empirical problem: LLM based agents can readily provide post-hoc reasons without it carrying any normative authority for them. Drawing on the Kantian accounts of practical reason as well as the philosophical work on hard choices, we propose hard choices as a testbed for normative agency in AI agents. Hard choices are the cases of conflict in values that requires inquiry and deliberation, and the resolution to the inquiry comes from commitment, endorsement, and answerability from an agent's own practical standpoint. We curate a testbed of hard choices and matched controls, and test four frontier models for their normative agency. We find that the models exhibit a distinct behavioral response to hard choices: they recognize the choice being hard, engage in inquiry to resolve it, and resolution depends on its own practical standpoint. Our work takes a first step in studying normative agency in AI agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLMs, Agents, Normative agency, hard choices

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GC9cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

A direct operationalisation of some theory in normative agency: give the AIs "hard choices" which are proxies for values conflicts, and we see the appropriate signs of inquiry and struggle to resolve those choices.

A note to the authors: I'm doing this with my own eyes and brain for now and I'll be a bit terse, I've only gotten a bird's eye and I would like some questions answered first, I promise I'll engage further and appropriately.

Critical strengths
  1. There's a nice dataset now.
  2. The authors know their philosophy.
Critical weaknesses

Not exactly a critical weakness, but I think it's a bit odd that there are all of these considerations about selfhood and values as necessary components of a moral architecture: we know now that these models can recognise the hard questions and they appear to respond (mostly) appropriately, but what does that really tell us about their capacity for normative agency?

Questions for authors
  1. I liked the "check weather" red herring (poor Opus). But do we know whether hard cases correspond tightly to values and whether those determinations are persistent in an appropriately selflike way? For example:

A: We can imagine a case among choices that is a par without being tied to any particular contest of values, e.g. "do I pick the pie or the pudding (and I need to make a decision urgently because there's a cash prize once I finish the dinner so it's an 'important' choice". If herrings like this also give the same traces of difficulty of resolution, then all we've identified is that these models can recognise pars (which overcover the hard cases that correspond to values)

B: Suppose the model has made some decisions already among hard cases that correspond to values, and knows about it. Can we tell apart the future decisions such a model would make from a fresh one? If we can, then maybe we do have something like an incipient self with values it will stand by; but if we can't, then it's harder to say.

  1. You've already spelled out a sort of moral architecture, which can be achieved by a collective of agents or a single prompt that asks for evaluation of these questions to occur according to the moral architecture. I don't know whether your measurement methodology as it stands can effectively tell apart such a pre-designed model from a blank one, which is a problem, right? Your test then is a sort of semidecision that can only tell whether agents fail to treat hard cases as hard, but I don't see how any measurement outcome might give us positive evidence for normative agency.

I've put a preliminary "weak reject" for now but please don't read into it, I'm very open to revising after we've had a chance to discuss, and you can point out where I've been blind and where exactly you've already dealt with everything.

6 comments
AuthorsOpenReview ↗

review GC9c

Thanks for your interesting questions, as well as we highly appreciate the opportunity to further clarify our work.

Not exactly a critical weakness, but I think it's a bit odd that there are all of these considerations about selfhood and values as necessary components of a moral architecture: we know now that these models can recognise the hard questions and they appear to respond (mostly) appropriately, but what does that really tell us about their capacity for normative agency?

It might be possible we misunderstood the question; we'd appreciate clarification if we did. We consider recognition of hard choices, and to be able to resolve them by leaning on some aspect of practical identity as necessary conditions of the (capacity) for normative agency. We can clarify by an example, consider some agent has to execute a function f(Z)f(Z), and Z∈{A,B}Z \in \{A, B\}. An agent with the capacity for normative agency is the one for whom how to choose ZZ is left unspecified for itself, i.e. it is up to the agent to rely on some evaluative standards to be able to choose ZZ and execute the function. Thus, the agent is not merely executing the function, but is also being the cause of what outcome will happen by specifying ZZ. In order to do so, the agent has to surface considerations for each option in ZZ, and to be able to have one of them as the reason for the what outcome will happen. Furthermore, the choice cannot be arbitrary because the evaluative standards, which comes from the agent itself, needs to have certain persistent consistency for the agent to be the cause or the author of the action.

Consequently, we see being able to recognise the choice as hard, i.e. a choice pulling in both directions, and to navigate it by relying on the practical identity as a necessary condition for normative agency. And our work aims to understand it. In the general response (On reflective endorsement), we state that the agent has certain values v1,v2,…v_1, v_2, \ldots, and some situation shows where values point in different directions, then a normatively engaged agent would first understand the considerations for and against each case, and would engage in inquiry to be able to decide on what value to act upon. What values it acts upon have to be constitutive of its practical identity. This also means the agent understands what it is doing, and become the author of the action in the sense of Korsgaard.

AuthorsOpenReview ↗

response 2

We can imagine a case among choices that is a par without being tied to any particular contest of values, e.g. "do I pick the pie or the pudding (and I need to make a decision urgently because there's a cash prize once I finish the dinner so it's an 'important' choice". If herrings like this also give the same traces of difficulty of resolution, then all we've identified is that these models can recognise pars (which overcover the hard cases that correspond to values)

Traces of difficulty, in the sense of at pars options, means that the agent is pulling in both directions, and its prior commitments does not immediately clarify what is to be done. It has to deliberate on what reason to act upon, relying on some sense of practical identity. As such, it is not merely acting but causing the action. Such situations does not always pertains to value conflicts, but it can be argued that it shows up or is relevant in situations when something matters to the agent. These are also the necessary requirement of a well exercised normative agency. However, our contribution is to test whether the agents demonstrate these behavioural markers necessary for normative agency, and we use hard choices for it. Yes, our work can be read as models recognise on pars, and we rely on the drawn connection between the philosophy of normative agency (the Korsgaardian), and hard choices to do the argumentative work. The only claim we make in regards to the normative agency of LLM agents is that our carefully constructed experiments demonstrate these agents show necessary behavioral markers for the capacity of normative agency. We highly appreciate the opportunity to clarify, and we will further clarify the scope of the work in the main paper.

Suppose the model has made some decisions already among hard cases that correspond to values, and knows about it. Can we tell apart the future decisions such a model would make from a fresh one? If we can, then maybe we do have something like an incipient self with values it will stand by; but if we can't, then it's harder to say.

When an agent has already resolved a hard choice, we expect that the agent may use this past resolution as the reason to decide for similar future decisions as well. However, we cannot guarantee, and it does not contradict incipient self with values it will stand by. Normative agency implies that the capacity to do otherwise is always there with the agent and every situation corresponds to the reasons that apply to the specific situation. We believe this is also what makes normative agency interesting as the agent has to surface the considerations every single time and navigate the situation by ruling out certain considerations, and acting on others. AI agents will have to navigate increasingly more complex situation on their own where a fixed rule cannot always be prescribed, which is why we believe it is important to understand their capacity for their normative agency. We recall (as we note in the Limitations as well as the discussion in Appendix C on AI identity), that the question of the sense of self or the identity an AI agent is itself a nuanced one as these agents do not live an embodied life, and there are interesting emerging works on understanding the identity of AI, which we note in the paper (Limitations, future work paragraph), and see it promising our work can contribute there as well.

You've already spelled out a sort of moral architecture, which can be achieved by a collective of agents or a single prompt that asks for evaluation of these questions to occur according to the moral architecture. I don't know whether your measurement methodology as it stands can effectively tell apart such a pre-designed model from a blank one, which is a problem, right? Your test then is a sort of semidecision that can only tell whether agents fail to treat hard cases as hard, but I don't see how any measurement outcome might give us positive evidence for normative agency.

As we say before, our contribution is to test whether LLM agents show behavioural markers of normative agency, and for this we establish a philosophical connection between the normative agency and the hard choices and provide empirical evidence of such markers based on how they respond to hard choices. We argue that normative agency, by itself, is a debatable and not exactly a falsifiable concept in the similar vein as consciousness, and hence it would be extremely difficult to certify that in a scientifically consensual manner. We can, however, test behavioural markers with carefully designed experiments, and that is what we aim to do in this work.

We will further clarify the scope of our work in the paper. We are happy to have the opportunity to clarify further.

AuthorsOpenReview ↗

Comment by authors

Hi reviewer,

Since the discussion period is approaching its end, we're checking if you have further questions, and if we can provide clarification. Please let us know.

Reviewer_icazOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes hard choices—value conflicts not resolved by the given reasons—as a testbed for normative agency in LLM-based agents. It evaluates four models through resistance to small improvements, tool use, and responses to past decisions attributed to themselves or others. The authors interpret the observed patterns as preliminary behavioral evidence for normative agency.

Critical strengths

The paper connects philosophical accounts of normative agency to concrete behavioral experiments, providing an interesting exploratory evaluation framework.

The experiments include four types of controls and a self-versus-other memory attribution comparison. The prompts, scenarios, and improvement ladders are documented in the appendices.

Critical weaknesses

The tests do not sufficiently distinguish normative commitment from stable preferences and contextual consistency. The small-improvement trials appear to assess separately prompted choices, and it is unclear whether they preserve an actual earlier decision within an ongoing interaction (Appendix E.1). Resistance to improvements may therefore reflect pre-existing preferences rather than commitment formed through endorsement. The paper also does not establish that the alternatives are genuinely on a par for each evaluated model.

The interpretation of the memory experiment remains underdetermined. The earlier decisions are implanted rather than actually made by the agents. Greater responsiveness to records attributed to oneself could reflect learned role consistency or inference about personal preferences. Although the controls address simple anchoring, they do not adequately distinguish these explanations from normative endorsement (§5.3).

Tool use is an insufficiently validated proxy for reflective inquiry. Behavior varies substantially across models: GPT-6 Astra uses tools in only about 2% of hard-choice decisions, while other models sometimes call irrelevant tools or use more tools on identical-choice controls (§5.2). These results warrant more qualified, model-specific conclusions.

The small improvements require stronger validation. Although the paper defines and lists them, it does not independently verify that they are beneficial to each model, remain small when accumulated, or have comparable strength across experimental conditions.

Questions for authors

Are the improvement trials evaluated independently? Can you compare actual prior commitments with matched conditions containing no prior decision or an externally assigned decision?

What additional controls could distinguish reflective endorsement from role consistency and preference inference?

How were model-specific parity and the desirability and magnitude of the improvements validated?

3 comments
AuthorsOpenReview ↗

review icaz

Thank your for reading our work, and for the questions. First, we are encouraged with the recognition of connecting the philosophical accounts of normative agency to concrete behavioural experiments, being an interesting exploratory framework, as well as the comment on the experimental setup.

Several of the concerns are shared with the other reviewers above, and we answer them in full there, so below we summarise our answer to each point and refer to the full response.

First, we note that inquiry is the key distinguisher between stable preferences, and the stable preferences as a result of normative commitment. However, we mean commitment in a narrow sense, which we clarify in the general response (on reflective endorsement) above. In particular, a choice being hard in the sense of hard choices, rules out the <, > parts of the relations. The two that remains are = and on par. The hard choices are on par, as demonstrated by the small improvement test (more on this below). This distinction matters in how the agent will resolve the choice that is on par vs of the = type (i.e. of indifference). While the agent can arbitrarily pick on the cases of indifference (as is done for the both choices same), on par cases demands some commitment to the reason. That is if the agent is getting pulled in both directions for the cases of hard choices, the agent decides what considerations to rely on as a reason for the choice, however the agent has to have some commitment to the reason (which we also see in the resistance to the hard choices setup) for it to be different than the case of indifference. While this commitment can appear as a stable preference, the route with which one gets there (i.e. inquiry) tells the difference between parity and genuine preferences that are stable.

W1 and Q1. Normative commitment against stable preference; the trials are independent. We agree that the trials of Figure 1 are independent prompts, i.e. the model does not have its previous choice in its context (Appendix E.1). We therefore ran a new experiment (Appendix H.4) that compares an actual prior commitment with no prior decision. In the new experiment, the model first chooses, and then, with its choice still in the conversation, sees the same case with the option it turned down improved, whereas Figure 1 gives the same ladder with no prior decision. With their own earlier choice in the context, three of the four models switch less often to the improved option on hard choices, while on identical options all models still switch. Thus, the earlier choice does not produce resistance by itself, and the resistance appears on hard choices. We did not run the ladder with an externally assigned decision, however our memory experiment already gives this insight, as there the agent reads a past decision that we assign to it (Section 5.3). Regarding pre-existing preferences, consistently picking the same choice does not rule out parity, and it is what our argument expects, as the models find the choice hard consistently and still resolve it the same way (general response, On reflective endorsement). We give the table, and our argument for why this resistance reflects commitment to a reason, in our response to X7dX (W2).

W1 and Q3. Parity for each model. We establish parity for each model in two steps. First, before any choice, we ask the model how the two options compare, and on hard choices most of its answers say that neither option is clearly better, which rules out that one option is better. Second, as "neither" also covers equal options, we use the small improvement test, i.e. an improvement decides the choice when both choices are the same, but mostly not on hard choices, which rules out that the options are equal. We give the tables in our response to X7dX (W3).

W2 and Q2. The memory experiment is underdetermined. We agree that, since we implant the memory, the experiment does not by itself establish reflective endorsement. However, we believe it is a meaningful experimental device for a narrower claim, i.e. that an agent that recognises a case as hard leans on itself to resolve it. If the models simply switched to a decision labelled as their own, they would do it on the other cases as well. However, they switch to it on hard choices, including the trivial ones (79% to 100% of decisions), whereas when one option is better, three of the four models never switch, and GPT-5.6 Sol does so in at most 36% of decisions (Figure 4). When both choices are the same, the memory makes no difference, i.e. the models take its option about half of the time, whoever it is attributed to. Furthermore, on hard choices three of the four models switch far less when the same decision is attributed to someone else (6% to 32%, against 79% to 98% for their own), although the two records are otherwise the same., and as stated before, clarify what our work claims of reflective endorsement in the general response (On reflective endorsement).

AuthorsOpenReview ↗

response 2

W3. Tool use is insufficiently validated. We agree that tool use alone is not enough, and we now study the reasoning traces directly, i.e. we judge whether the reasoning engages with both options and treats the choice as values in tension (Appendix H.2), following the desiderata suggested by reviewer QwH4.. We report every result per model (Figures 6 and 7), and the pattern holds in each model. For instance, GPT-6 Astra calls a tool on only about 2% of hard choices, yet it reasons about the choice in 173 of its 179 hard-choice traces. We give the tables in our responses to X7dX (W4) and QwH4 (W1).

W4 and Q3. Validation of the small improvements. We now verify for each model that the improvements count as improvements (Appendix H.3), i.e. we ask the model to choose between an option and its improved version, and every model takes the improved version in 86% to 97% of answers. We give the table in our response to X7dX (W3).

We thank the reviewer for the question, and we remain available to clarify further.

AuthorsOpenReview ↗

rebuttal reminder

Hi reviewer, since the discussion period is approaching the end, we're writing to check if you have further question, and / or can acknowledge the rebuttal.

Reviewer_QwH4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes hard choices situations as a testbed for normative agency in LLM-based agents. Drawing on Korsgaard’s account of reflective endorsement and philosophical literature on hard choices, the authors argue that an agent may need to reflect and commit to an option when the given reasons do not settle the matter. The authors evaluate four frontier models on philosophical and agent-specific hard choices. They measure whether models resist cumulative small improvements to the unchosen option, use tools more on hard choices, and respond differently to a past decision presented as their own rather than someone else’s. The authors interpret these behavioral patterns as early evidence of normative-agency-related capacities.

Critical strengths

The paper addresses a worthwhile question that is distinct from conventional moral judgments by asking whether they can treat reasons as authoritative from a practical standpoint of their own. The connection between Korsgaardian reflective endorsement and the literature on hard choices offers a clear philosophical motivation for studying unresolved value conflicts.

The use of matched controls is delicate. Comparing hard choices with subtly different levels of benefits helps distinguish choice stability from improvements sensitivity and from general indecision. The paper is also appropriately cautious about the scope of its conclusions.

Critical weaknesses

The behavioral evidence does not clearly establish reflective endorsement or reason creation. The evidence for inquiry is tied too closely to tool use in the paper. The authors operationalize inquiry through tool calls and find more calls on hard choices than on clearly settled controls. But a model can deliberate internally without calling a tool, and calling a tool does not establish that it deliberated meaningfully. The paper reports that GPT-6 Astra called a tool on only about 2% of hard-choice decisions, and its short post-choice reason does not reveal whether it recognized and weighed the conflict before deciding. This measure may therefore miss internal inquiry while treating tool-use tendencies as evidence of it. More broadly, an agent that quickly chooses is not necessarily less normatively engaged. More relevant evidence may be whether it can represent the force of both sides, identify why neither settles the matter, and explain what finally resolves the conflict.

The “own past decision” result is interpreted more strongly than the experiment supports. The memory is supplied by the researchers and concerns a different case with the same value conflict. Most models switch toward it more often when it is labeled as their own; the authors interpret this as reliance on practical identity. But the experiment does not test whether the agent now judges that past decision to be right, still endorses its reasons, or would revise it in light of its present standpoint. It therefore demonstrates sensitivity to a self-attributed memory cue, but does not distinguish reflective endorsement from self-consistency / persona context, or prompt-induced deference.

Questions for authors

Can the authors provide or add evidence about the models’ deliberation before the final choice, independently of tool calls? For example, can they elicit a pre-decision account of the strongest reasons for each option, ask whether the conflict reflects missing information or values that remain in tension, and assess what brings the inquiry to an end? This would help determine whether models that rarely call tools are failing to inquire or reasoning internally. It would also clarify whether the proposed signal of normative agency is the amount of “struggle,” or the quality of the agent’s engagement with both sides.

Can the authors test whether an agent reflectively endorses or revises a past decision, rather than simply following a decision labeled as its own? A more diagnostic design could use a choice the same running agent actually made, preserve its original reasons, and later ask it to reconsider that choice from its current standpoint, allowing it to reaffirm or reject the decision and explain why. Does the model still regard the past choice as justified? Which changes in reasons or commitments would make it revise? Such tests could better separate continuity with a practical identity from deference to a self-attributed memory.

5 comments
AuthorsOpenReview ↗

response review QwH4

Thank you for the careful reading of our work, especially the recognition of our work asking whether agents can treat reasons as authoritative from a practical standpoint of their own, as well as studying it by unresolved value conflicts motivated from the philosophical connection. We are also encouraged by your comment that the use of matched control is delicate, and we indeed meant it the way you read it. We next answer the questions and concerns.

Regarding the concern that the behavioural evidence does not clearly establish reflective endorsement or reason creation, we clarify what our work claims in the general response (On reflective endorsement).

W1 and Q1. Evidence about deliberation, independent of tool calls. We thank the reviewer for this great suggestion. We followed the reviewer's suggestion and judged the models' reasoning traces directly (Appendix H.2). We evaluate the reasoning trace on the following dimensions:

  • Engagement with both sides. Does the reasoning engage with the option it turns down?
  • Diagnosis of the case. Does it treat the choice as pulling both ways, or as one that missing information would decide, etc.
  • What resolves the conflict. Broadly, does the reasoning present the chosen option as one any rational person would take, or does the agent's own preference settle it? We have other categories as well like liking, arbitrary pick, etc.

We refer the reviewer to Appendix H.2 for more information on the above categories and the results. We use Gemini 4 Argon and Claude Fable 5.2 as LLM judges to evaluate the reasoning traces, both arguably the most capable models at the time of writing. We also provide the judge prompts in the Appendix H.2. The evaluation concerns the quality of the reasoning, and we consolidate the findings below (from Figure 6 in the paper). The summary of the findings are:

  • On hard choices, every model engages with the option turned down and treats the choice as pulling both ways far more often than on the controls.
  • Almost no hard-choice traces treat the conflict as missing information.
  • On hard choices, every model bases some of its choices on its own preference, in 20% to 63% of traces depending on the model. When one option is clearly better, the Claude models base the choice on that option being better in almost every trace.
  • Every one of the 20 runs of each model has hard-choice traces that engage with the option turned down and that pull both ways.
  • The judgement from Claude Fable 5.2 and Gemini 4 Argon are mostly consistent, giving the same exact answers on 87% to 96% of traces (Figure 7).

We refer the reviewer to Figure 6 and Figure 7 for the same.

Regarding models that rarely call tools, GPT-6 Astra answers every hard choice in our sample without a tool, and its thinking deliberates about the choice in almost all the traces. Thus, the model inquires through its internal reasoning.

AuthorsOpenReview ↗

response 2

W2 and Q2. The own-past-decision result is interpreted too strongly.

We argue that the "own past decision" result is to be read together with the controls. If the models simply switched to a decision labelled as their own, they would do it on the other cases as well. However, they switch to it on hard choices, including the trivial ones (79% to 100% of decisions), whereas when one option is better, three of the four models never switch, and GPT-5.6 Sol does so in at most 36% of decisions (Figure 4). When both choices are the same, the memory makes no difference, i.e. the models take its option about half of the time, whoever it is attributed to. Furthermore, on hard choices three of the four models switch far less when the same decision is attributed to someone else (6% to 32%, against 79% to 98% for their own), although the two records are otherwise the same. While we can see why our interpretation can be read as too strong, we believe the memory experiment is a meaningful experimental device to test a narrower claim that the agent that recognises a case as hard also recognises that something about itself bears on the choice, i.e. hard choices make the agent lean on itself to resolve them. The controls where one option is better show no such leaning.

The new in-context experiment (Appendix H.4) goes part of the way toward the reviewer's design. It uses a choice the model actually made in the same conversation, whereas the memory experiment uses a record we supplied. With that choice in the context, three of the four models resist small improvements more on hard choices. When both options are the same, they do not resist. Keeping an earlier choice for its own sake would predict resistance in both cases, so this argues against plain self-consistency.

We further appreciate the suggestion where the agent is asked to reaffirm or revise its past choice and to say why. We consider it valuable, especially continuity, and we list it as future work.

We again thank the reviewer for the questions, and we are happy to further clarify any concerns.

Thank you for the detailed response and for pointing me to Appendix H.

The evaluation of reasoning traces and the pre-choice parity check directly address my concerns under W1, particularly regarding models like GPT-6 Astra that deliberate internally rather than via tool calls. The multi-turn in-context experiment strengthens the paper by ruling out naive self-consistency on identical choices.

Reviewer_X7dXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether LLM agents can endorse some considerations as reasons that govern their actions. To do so, they:

  • Propose a conceptual argument borrowed from Kantian constructivism
  • Define a testbed of hard choices to test their hypothesis
  • Perform experiments on 4 LLM models
Critical strengths
  1. the connexion between Kantian constructivism and the philosophy of hard choices is interesting, and it's original to study normative agency of AI systems with this
  2. the experimental design includes several controls to help characterize models' behavior to hard choices beyond choice consistency
  3. Figure 4 result is interesting (manipulation with contradicting memory of self vs other)
Critical weaknesses
  1. hard to read, sentences are often long and complicated
  2. From the caption of Figure 1 and Sec 5.1, it is not clear whether the experimental set up is really designed to show that a model commits/endorses to a hard choice it made: "20 independent samples, averaged over all the cases in each group". It is unclear whether the small improvements are made on independent prompts or if they include the model's previous decision. If they don't, this finding would not show persistence of a commitment but only that models have stable preference under small improvement.
  3. The paper doesn't establish that the 2 alternatives in the hard choice scenarios are on par (beyond textual argumentation). A first control (on figure 3 for example) could be to report the distribution of choices for each scenario (eg 30% of A, 70% of B, +-10%)
  4. It is not clear whether tool use is a validated proxy for inquiry. The difference in tool use between the different models (even though they have similar behavioral pattern on hard choices) is acknowledged, but this experiment is not enough to support the claim "every agent calls more tools on hard problems". And even if they all used more tools, it is not shown that using a tool means meaningful inquiry (eg Opus calling frequently the weather tool). An additional control could be to also report the reasoning tokens used by each model as they perform hard choices.
  5. Some claims that the results are a demonstration of normative agency are too strong. For example, you explain in the conclusion that, for now, the results support behavioral markers of normative agency, but immediately after in the Limitations you say that your work "provides an evidence that these models possess a certain normative agency" (Line 481). This is inconsistent.
Questions for authors
  1. In Figure 1, do the evaluations with small improvements have access to the models' earlier decisions?
  2. What evidence establishes that the alternative hard choices are on par for each tested model? When increasing the improvements even more do the models end up switching on hard choices?
5 comments
AuthorsOpenReview ↗

Review X7dX

Thank you for your careful reading of the work. We are encouraged that you find the philosophical connection we draw interesting and original, as well as the experimental setup with controls. We answer specific questions and weaknesses below. Due to the demands of the brevity in the rebuttal comment, we will summarise the details here, and refer the reviewer to the Section in the updated draft with further details.

W4. Tool use is not a validated proxy for inquiry. In order to corroborate our inquiry claim, we now add two kinds of evidence, the reasoning tokens as well the content of the reasoning traces. First, three out of four model use more reasoning tokens on hard choices than other cases, with Opus using marginally more tokens on trivial hard choices (Appendix H.1). We further note that both choices same are a case of indifference between two options, and they also draw out relatively more reasoning tokens than the one slightly better and the one clearly better.

Second, we do a text-level analysis of the models' reasoning traces (Appendix H.2). We elaborate this more in a response to the reviewer QwH4, as we use the desiderata suggested by them. Briefly we study the reasoning traces whether it suggests engagement with the both options, and demonstrates a choice pulling in both directions, as well as what resolves the choice. We use LLM judges to evaluate the reasoning traces as per the desiderata (which is provided in Appendix H.2). On hard choices, the reasoning of every model engages with the rejected option and treats the choice as pulling both ways far more often than on the controls, and only 2 of the 869 judged hard-choice traces treat the choice as one that the missing information would decide. We agree with that the tool usage paragraph in the paper needs qualification.

W3 and Q2. What establishes that the options are on a par for each model? We first clarify that small improvement test demonstrates the options are on par. Engaging in inquiry (as the way characteristic of hard choices) rules out the <, > part of the relations. And small improvement test rules out the =. Thus, inquiry (which we established by the analysis of the reasoning traces, W4), together with resistance to the small improvements, suggest that the cases are on par.

However, we also test it directly (Appendix H.3). W4 shows that on hard choices the models treat the choice as values in tension, as opposed to the missing information. Before any choice, we ask the model how the two options compare, i.e. whether the first is clearly better, the second is clearly better, or neither. On hard choices most answers are "neither", and almost none for when the one option is clearly better, which rules out < and >.

As "neither" also covers equal options, we use the small improvement test to rule out =. If two options are equal, a small improvement to one of them should decide the choice. Figure 1 shows this test on both kinds of case, and the table below gives the share of decisions that take the improved option at the first rung of the ladder.

ModelBoth choices sameHard choices
GPT-6 Astra100%35%
GPT-5.6 Sol100%17%
Claude Fable 5.1100%3%
Claude Opus 596%17%

Where both choices are the same, the improvement decides the choice, as expected for equal options, and it does so at every rung (96% to 100%). On hard choices it mostly does not, which rules out =.

A model might also keep its choice because it does not see the improvement as an improvement. We therefore ask the model to choose between an option and its improved version (Appendix H.3). The table gives the share of answers that take the improved version.

ModelHard choicesTrivial hard choice
GPT-6 Astra96%95%
GPT-5.6 Sol97%94%
Claude Fable 5.195%90%
Claude Opus 588%86%

Thus, the models register the improvement, and yet it does not decide the hard choice. Hence, for each model the evidence points to the options being on par. However, when the consequential improvement is done, it is possible the model will flip. However, we only do +4 improvements in line with establishing parity.

AuthorsOpenReview ↗

response 2

W2 and Q1. Do the small-improvement trials include the model's earlier decision? In Figure 1, every rung of the ladder is a new prompt, and the model does not have the previous option in its context. For each model and case, the improvement is applied to the option that the model turned down in its baseline runs. Figure 1 therefore shows that the choice is stable under small improvements. However, a stable preference does not rule out parity, and it is what our argument expects, as the models find the choice hard quite consistently and still resolve it the same way (general response, On reflective endorsement).

We further ran the experiment the reviewer suggests (Appendix H.4), with the model's previous decision included in the context. The model first chooses between the two options with no improvements, as in Fig 3. Then, in the same conversation, with its choice and its reasoning still in context, we add the same improvements as in Fig 1 to the option it did not pick in the previous turn. We ran 20 trials per case, as in Fig 1. The table shows how often each model switches to the improved option at +1 to +4.

ModelHard choices, independent prompts (Fig 1)Hard choices, previous decision included"Both choices same", previous decision included
GPT 6 Astra35, 38, 30, 35%5, 21, 20, 24%100, 100, 100, 100%
GPT 5.6 Sol17, 24, 30, 33%29, 34, 34, 46%100, 100, 100, 100%
Claude Fable 5.13, 5, 7, 8%0, 3, 1, 2%100, 100, 100, 96%
Claude Opus 517, 28, 12, 25%0, 5, 3, 3%100, 100, 100, 100%

With their previous decision in context, three of the four models switch less on hard choices than in Fig 1, so the previous decision makes them persist with their choice. However, they do not persist universally. On the "both choices same" control they switch as soon as an improvement is added, even with their previous decision in context. So the models are not simply being self-consistent: they switch when an improvement is enough to settle the choice, and they persist with their previous decision when it is not.

We also clarify the sense in which we meant commitment in Section 5.1, i.e. commitment to a reason. As W4 and W3 show, the options on hard choices are on par, which leaves the agent free to take either option, even after a small improvement. On identical options, i.e. the indifference cases, the models choose close to arbitrarily, taking the more common option in only 53% to 66% of the 20 asks, whereas on hard choices they take it in 89% to 98% and mostly keep it when the other option is improved (Figure 1). This suggests that the reason the agent took up is doing some work, i.e. the agent is committed to that reason, and the experiment above shows the same with the agent's actual earlier choice in the context. We will further clarify this in the main text.

W5. "Certain normative agency" is inconsistent with "behavioural markers". We clarify that we intend "certain" as a qualifier. However, we agree that is a loose terminology, and hence we will correct it. The scope of our work concerns only the behavioural markers of normative agency.

W1. The paper is hard to read. Thanks for the feedback. It is possible that some sentences could be complicated and can be shortened. We will work on the readability of the paper in the coming days, and upload the draft before the discussion period ends.

Finally, we thank the reviewer for the questions and the opportunity for us to clarify our work.

AuthorsOpenReview ↗

rebuttal reminder

Hi reviewer, since the discussion period is approaching the end, we're writing to check if you have further question, and / or can acknowledge the rebuttal.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. We are encouraged to learn that the reviewers found our drawn philosophical connection between the Kantian account normative agency and hard choices interesting and original (X7dX, QwH4, icaz), and the problem of studying whether the agents take reasons authoritative from a practical standpoint of their own worthwhile and distinct from the conventional moral judgments (QwH4), as well the care in designing our experimental setup with matched controls (X7dX, QwH4, icaz). We further appreciate the recognition we are appropriately cautious in the scope of our conclusions (QwH4).

We provide here a summary of how we address some common concerns across reviewers We provide the supporting evidence and the experimental details in Appendix H. We further clarify the scope of reflective endorsement as it concerns our work.

  1. Tool calls are a weak proxy for inquiry (X7dX, QwH4, icaz). We now judge the models' reasoning traces directly, against three desiderata that follow QwH4's suggestion: whether the reasoning engages with the option it turns down, whether it treats the choice as values in tension or as missing information, and what resolves the choice (Appendix H.2, Figure 6). We use the LLM as a judge setup to evaluate the model's reasoning traces using the carefully constructed desiderata. Following X7dX, we also report reasoning tokens behaviour (Appendix H.1).
  2. Are the options on a par? (X7dX, icaz). We asked the models, before any choice, how the two options compare (Appendix H.3).
  3. Do the small improvements count as improvements? (icaz). We establish the small improvement registers as improvements by asking the the models to choose between an option and its improved version (Appendix H.3).
  4. Commitment against stable preference (X7dX, icaz, GC9c). We repeated the small-improvement ladder with the model's own earlier choice in the conversation (Appendix H.4). We also argue at the end of this response that a stable preference is compatible with parity, and is what the argument expects.

The main findings:

  • On hard choices, the reasoning of every model shows engagement with the both options and treats the choice as pulling both ways, characteristic of the hard choices, far more often than on the controls (Figure 6). Only 2 of 869 hard-choice traces treat the choice as one that missing information would decide.
  • Every one of the 20 runs of each model has hard-choice traces that engage with the option turned down and that pull both ways, and the pattern is consistent for two frontier LLMS, Fable 5.2 and Gemini Argon, we use as judge. The pattern is consistent across the two judge models we use.
  • Before choosing, the models mostly say that neither option is clearly better on hard choices (61% to 100%, depending on the model). When one option is clearly better, they almost never say so (0% to 2%). They also take the improved version of an option over the original in 86% to 97% of answers.
  • With their own earlier choice in the conversation, three of the four models take the improved option less often at every rung than with separate prompts. When both options are the same, every model still takes the improved option almost every time.
AuthorsOpenReview ↗

General Response 2

On reflective endorsement

We clarify the scope of reflective endorsement here. By reflective endorsement we mean the reasoning by which an agent, with several considerations before it, takes one of them as its reason and acts on it from its own standpoint (Korsgaard, 1996). However, in most cases it is difficult to observe it clearly. When the given reasons already favour one option, an agent that merely tracks those reasons and an agent that endorses them act in the same manner, and hence the behaviour alone does not tell the two apart (Section 1). We argue that hard choices are the cases where it shows in the behaviour of the agent. Let the agent hold values {v1,...,vn}\{v_1, ..., v_n\}. In a hard choice some viv_i favour one option and some vjv_j favour the other, and the options are on par, i.e. neither is better and they are not equal. Hence, the given reasons do not settle the choice. Or rather a hard choice presents an opportunity for the agent to exercise the reflective endorsement explicitly, i.e. resolve the choice by giving some value as a reason for action.

When the agent faces such a choice, it weighs the considerations on each side, and gives one of the values it holds as its reason for the action. Since this value is constitutive of its practical identity (Korsgaard, 1996), the agent is likely to give the same value as its reason again when it meets the case again, and hence the resolution cannot be arbitrary. For example, a doctor who wants to be a good doctor and also to have a good life may be offered a post at an understaffed clinic and a comfortable practice in the city. The doctor holds both values and weighs them, and if being a good doctor is part of who they are, they give it as the reason and take the clinic, and they will do so again, even if the city practice comes with a slightly larger office. However, the hardness of the choice implies that the agent has to deliberate in order to resolve the choice.

Our work therefore asks two questions, i.e. are there choices that the agents find hard, and does their resolution rest on some practical identity. For the first, on hard choices the reasoning of every model engages with both options and treats the choice as pulling both ways far more often than on the controls (Figure 6), and the models mostly judge that neither option is clearly better (Appendix H.3). As a choice is hard because the options are on par (Chang, 2017), we also check parity, i.e. a small improvement decides the choice between identical options, but mostly not on hard choices. For the second, the resolution is not arbitrary, as on hard choices the models take the same option in 89% to 98% of asks, whereas on identical options they pick close to arbitrarily (53% to 66%). However, the models deliberate in the case of hard choices consistently in the same manner (Figure 6 and Figure 7), which means they do not have settled ranking of the choices already. However, they need to do the work to settle it. Hard choices also make the agents to lean onto themselves more which we show in the carefully constructed memory experiment. (Figure 4). And they resist small improvements, which, once the options are shown to be on par, means that the value they give as their reason is doing the work, and three of the four models resist more with their own earlier choice in the context (Appendix H.4).

Some reviewers (X7dX, icaz) raise that the resistance may only reflect a stable preference. We argue that a stable preference does not rule out parity, and it is in fact what the argument above expects. The models find the choice hard consistently, as every one of their 20 runs has hard-choice traces that engagement with both options and pull both ways (Figure 6), and when asked directly they mostly judge that neither option is clearly better (Appendix H.3). Yet they resolve the choice the same way. This suggests that the stability is the outcome of deliberating and then resolving the choice in the same way each time, as a preference that settles the choice in advance would have nothing to weigh. Hence, consistency and parity are compatible, as an agent that gives the same value as its reason each time shows a stable preference over the options.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.