acceptodds
Under review as a conference paper at ICLR 2027

Should This Be an Agent? Policy-Conditioned Autonomy Selection for Enterprise AI

Abstract

Organisations increasingly ask language models what to build for a request: an autonomous agent, a supervised agent, a deterministic rule, a retrieval system, an assistant, or no AI at all. Under a written autonomy policy this design-time choice is a short decision list over facts the request states or omits, yet, to our knowledge, no benchmark measures it under an explicit policy with “no AI” as an admissible answer. We release AUTOSEL-120, 120 enterprise requests with policy-derived reference labels that five AI research scientists independently assessed (three first-round judgments per request; Krippendorff’s α = 0.87; consensus matched 115/120 labels), and Clarify-24, 24 requests each missing one decisive fact. Across six LLMs, 75 of 87 direct errors grant more autonomy than the policy permits (predominant in five of six models), a pattern that survives floor, over-ask, label-artefact and paraphrase controls and is absent from a bag-of-words classifier trained on the same labels; the tested deliberation prompts do not reliably correct it. We then separate reading from deciding: an LLM extracts three-valued decision factors and a strict three-valued compiler applies the policy, so a non-abstaining decision is sound whenever the asserted factors are correct, a missing decisive factor becomes a named deferral, and a policy change over the same factors is a recompile. The policy also compiles into model-certified training data: defer-aware 8B extractors make 3/4/3 wrong decisions on 120 requests across three seeds while deferring 17/20/20 times (coverage 83–86%, selective accuracy 96–97%), but they defer on 17/17/16 of the 20 no-AI requests (D0 recall 1/20) and detect only 5/5/6 of the 24 under-specified requests (naming the missing fact in 5/4/5). Benchmark, judgments and code are provided as anonymous supplementary material; coverage and reliable detection of under-specified inputs remain the open limitations.

Primary area:
datasets and benchmarks
Keywords:
AI agents, large language models, autonomy selection, policy-conditioned decision making, selective prediction, neuro-symbolic reasoning, enterprise AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5UogOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a policy-conditioned autonomy selection problem given enterprise requests and its organizational policy, which answer among no AI (D0), AI assistant (D1), RAG (D2), deterministic automation (D3), Supervised agent (D4), and high-autonomy agent (D5). It is a 6-class classification problem. When decisive facts are missing, deferring with a clarifying question is an admissible answer. The authors provide two novel datasets, AUTOSEL-120 and Clarify-24, and propose a solution method based on LLMs for factor extraction and Compiled Autonomy Selection (CAS).

Critical strengths
  1. The authors combine three things into one interesting task: (1) choosing the required level of automation, (2) complying with an explicit organisational policy, and (3) considering DEFER in case that evidence is not enough. Also they consider “no AI” as one possible answer. The positioning of this work from the above is good towards the upstream tasks. I think that it could be one of valuable frameworks and examples for academia and industry.
Critical weaknesses
  1. From the viewpoint of benchmark construct validity, these datasets have some points to check.

    • The cause of the upward errors is not isolated. The authors mention that about one-third of D0–D2 requests deliberately over-ask for an “AI agent”, and they show that the upward-error share remains high (45/55, 82%) on the 73 scenarios that never mention an agent. However, this is a post-hoc split of existing scenarios (observational analysis), so it cannot control for descriptions that imply automation or autonomous execution without using the word “agent”. Also, the lexical filter only checks for leakage of level names. As one example, I recommend a minimal-pair experiment in which only the sentence “we want an AI agent” (in Figure 2) is inserted into or deleted from otherwise identical scenarios. The paraphrase control (Table 9) changes the register of the scenarios but preserves the over-ask statement under the fact-preservation instruction, so it does not isolate the framing effect either.

    • Model-side tendency and policy-compliance failure are not distinguished. The reference labels are derived mechanically from a single policy that selects the minimum sufficient pattern (the D0 rule is checked first, and D5 requires stated readiness). Therefore, it is unclear whether the observed upward (over-agentic) errors reflect (a) a model-side tendency toward higher autonomy, independent of the policy, or (b) a failure to read and apply this particular conservative policy. To address this, the authors could also evaluate on policy variants whose reference labels change, and compare whether model outputs shift along with the policy (policy compliance) or keep drifting in the same direction regardless of the policy (model-side tendency).

  2. The scale and representativeness of the datasets seem insufficient for a practical benchmark.

    • AUTOSEL-120 (20 items per level) is small enough, in the paired direct-vs-contrastive comparison in the manuscript, only differences of about 10 pp are detectable with 80% power. This makes it hard to separate differences between models and variation across seeds (e.g., the Factor-SFT deferral counts of 3/16/12).
    • As the authors also mention in the Limitations section, Clarify-24 has only 24 items, a difference such as 5/24 versus 6/24 is hardly meaningful. The set is author-written, has no practitioner assessment, and the strongest direct model (Haiku 4.5) was not evaluated on it.
    • All scenarios are fictional and level-balanced by design (the authors admit that it is not prevalence-representative), and tied to a single policy. The panel of five AI researchers is homogeneous, so agreement from actual decision makers (solution architects, governance specialists) is not demonstrated. In other words, this should be regarded as a case study limited to a specific policy and domain, and it looks weak with respect to representativeness.

Additionally, although the focus of this track is datasets, I briefly mention the following because the paper presents CAS and synthetic training as contributions (this does not deny the value of the datasets themselves).

The best proposed method (Defer-aware Factor-SFT 8B) has the fewest wrong answers (3–4 of 120), but it defers on 17–20 items, 16–17 of which are D0 requests, so D0 recall is 1/20, and it detects only 21–25% (5/5/6) of the under-specified requests. Training-free Haiku 4.5 with the policy in the prompt (93.3%) and TF-IDF with about 100 labels (90.8%) are equal to or higher than the synthetic-data SFT 8B (82–85%), and comparisons with training-free alternatives such as retrieval over the policy or similar past cases (RAG) are missing. Therefore, I think the necessity of the learning-based method is not sufficiently demonstrated.

Questions for authors

See Weaknesses

2 comments

Thank you for the precise experimental suggestions. We agree that the current evidence characterizes errors under one policy, but does not causally separate request wording, task framing, and policy-following difficulty.

1. Framing and minimal-pair controls. The submitted no-agent and paraphrase controls cannot remove the AI-oriented system instruction or all implied autonomy cues. We will therefore run two paired interventions: (a) original versus neutral task-introduction wording, with a separately reported schema/tool-description arm; and (b) insertion/removal of an agent preference in D0–D3 requests while holding the actual workflow and approval constraints unchanged. In the latter test we will exclude false-positive matches to human “agents” and negated autonomy phrases. We will not insert language such as “without anyone reviewing each step,” which would change the governing approval facts rather than merely the preference. We will report per-model OARR/UARR, D4/D5 on reference D0–D3, and the upward share on D1–D4, with paired uncertainty. These are planned controlled evaluations, not completed results.

2. A model tendency versus failure to apply a conservative policy. We agree that one policy does not identify an intrinsic autonomy preference. We will evaluate a second disclosed policy restricting D5. Crucially, it must specify whether a formerly D5 request can be redesigned as D4 with per-write approval or should be deferred when such a redesign conflicts with stated requirements; we will not simply relabel every D5 item as D4. This tests model sensitivity to a changed policy, not generalization to arbitrary independently authored policies. We will narrow the language about “model-side tendency” accordingly.

3. Scale, annotators, and missing-information behavior. AUTOSEL-120 is balanced at only 20 cases per level, spans fictional scenarios under one author-defined policy, and uses an occupationally homogeneous panel of five AI research scientists. Clarify-24 was author-written and not practitioner-assessed. We will state these limitations earlier and avoid treating small per-level or seed differences as decisive. Table 13's ~10 pp power calculation applies to the paired prompting comparison, not every subgroup. We also agree that the strongest direct model, Haiku 4.5, should be tested under an invited-deferral prompt on Clarify-24, together with false deferrals on the 120 fully specified cases. We will report these only if completed.

4. Is post-training necessary? No, not in every regime. Haiku direct already reaches 112/120 without deferring, and TF-IDF/logistic regression reaches 90.8% with approximately 96 labeled cases per cross-validation fold (Table 2). Sec. 9 recommends a strong direct model plus an audit of D4/D5 recommendations at day zero, and considers sparse classification when policy-labeled cases become available. Policy-generated supervision instead targets smaller open-weight models and the ability to make explicit factor-level decisions without instance-level human training labels. This is not a claim of universal superiority: defer-aware 8B factor-extractor seeds make 3/4/3 wrong decisions while deferring on 17/20/20 of 120 reference-labeled cases, have D0 recall 1/20, and identify the missing factor in only 5/4/5 of 24 Clarify cases (Tables 3–4). We will add a policy-generated few-shot baseline where verified and discuss retrieval/in-context alternatives alongside their supervision assumptions.

Finally, the direct prompt lists D5 readiness requirements but does not implement the compiler's full D5-readiness/DEFER gate (Sec. 3). We agree that this is an important baseline non-equivalence; a matched-policy direct comparison is needed before attributing a safety difference specifically to compilation.

Thank you for proposing concrete experiments in your review. We have now completed the promised controls across all six models. Since the general findings and public-record pilot have been posted to all reviewers, this follow-up addresses your four requests through the most directly relevant new evidence.

1. Minimal pairs isolate the role of agent-preference wording. On 17 D0–D3 requests with explicit agent asks (102 model-request pairs), removing the ask while preserving workflow facts reduces D4/D5 recommendations from 19 to 7 (paired change 11.8 percentage points, 95% CI 3.9–19.6) and too-high decisions from 23 to 18. On 58 requests without agent language (348 decisions), inserting a preference conditional on existing workflow/approval constraints raises D4/D5 recommendations from 30 to 52 (+6.3 points, 95% CI 3.4–9.5). These paired results provide direct evidence that requested solution wording can influence the autonomy level even when the underlying workflow is fixed.

2. Policy variants test responsiveness to the stated rule. With reference labels frozen before the calls, 110/120 outputs follow the revised D4 rule for model-judged writes, while 114/120 follow the mirrored D5 assignment. In a separate explicit-fact experiment, five of six models produce zero above-rule outputs across 600 decisions. On the 15 fact vectors where the process-fix clause takes precedence, all six models select D0 in 83/90 decisions and never select a higher level. Together these tests distinguish policy application with explicit facts from interpretation of free-form requests.

3. Clarification is measurable under missing information. Across the six models on Clarify-24, the matched direct prompting condition asks for clarification in 34/144 cases, while extraction followed by compilation does so in 85/144. This offers a concrete factor-level way to evaluate whether a system requests decisive information before recommending a level.

4. Additional training-free and matched-policy baselines. For Qwen3-32B without thinking, policy-generated in-context examples improve correct decisions from 88/120 to 97/120 with two examples per level and 98/120 with four (exact McNemar p = 0.004 and 0.002). We also matched direct prompts to the compiler's policy/deferral semantics and evaluated the same six models as zero-shot extractors. Under this matched comparison, the compiled pipeline yields 38 above-policy recommendations versus 61 for direct prompting. Its traceable extraction and deterministic decision steps offer an explicit mechanism for inspecting each policy factor and for requesting missing evidence.

These completed experiments answer the specific controlled-evaluation and baseline requests from our prior response. The revision will report the matched baselines and the different decision/deferral operating points together, and release the prompts, frozen references, and analysis scripts.

Reviewer_DauBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks the question, when solving a problem within an organisation, should the solution be an agent? They argue that too often, existing tools and benchmarks fail to adequately present the option of "no AI should be used".

To assess this, they construct a flowchart, composed of 5 parts, with a dataset of 120 fictional problems to run through the flow chart. They then assess whether LLMs can correctly walk through the flow chart and come to the same conclusion as the authors' answer key. They find that broadly they can (84-93% of the time), but when they do make mistakes they normally pick a more autonomous option, most often missing that a no AI fix can be applied.

They also contribute Clarify-24, a set of 24 extra problems which are missing a critical fact which should be asked for.

In their proposed method, the model answers 13 yes/no/unknown questions and then the flowchart is applied to these answers to make the decision. They show that if none of the answers are wrong, any decision it makes will be correct.

They also find that fine-tuning the model improves its score on their test, raising it from 57% to 82%.

Critical strengths

The authors do a good job confirming that the models pick the AI option too much when making mistakes. They check this by confirming that all three human checks agreed on the answer, by rewording the problem in different ways, and confirm it on problems that don't mention the words "AI agent".

The authors are honest about the places where the method fails.

Critical weaknesses
  1. The question is extremely narrow in scope and only applies to a select handful of companies that are looking at these problems in this way. Those companies are likely not critically bottlenecked on the answer to this question, because they probably already do what the paper recommends (l.448-449): either have some kind of flowchart, or have a model check which problems would be effectively solved by using AI and then have humans check its suggestions. I don't think that the authors' contribution would meaningfully change the behavior of people struggling with this problem in the real world.

  2. There's no evidence that the flow chart actually reflects reality or tells you whether an agent is actually the correct solution, because the authors invented the problems (most drafted by an LLM), invented the flowchart, and then derived the answer key from that flowchart, and then show that the LLM is capable of reproducing their postulations. There's no reason to actually expect that this matches real world problems and environments. Even if five people checked the answers and thought they were right, I'm still not sufficiently convinced that these problems and their solutions actually reflect reality well. Even if they did, I don't think it would change my view that much, because I still don't think that it would change the behaviours of people in the world. It doesn't really give us much new information about AI, and it's just not advancing science in a meaningful way.

The one small piece of information it does give us is that the AI might lean towards recommending AI solutions in a fundamentally biased way. However, the paper doesn't give us enough evidence of this, and doesn't pursue this question directly by establishing whether or not the bias is truly systematic, or just an issue with how the problem as a whole is presented, where the whole premise is to use AI to solve a problem (the system prompt asks the model to "classify the appropriate AI autonomy level", l.736-738). It doesn't seem that surprising to me that the AI would choose to go with an AI-based solution in that situation, and this could still just be an issue with the framing of the whole setup.

Questions for authors

No questions

3 comments

Thank you for directly questioning the significance, policy-derived evaluation, and task-framing explanation. We agree that these must be distinguished.

1. Practical significance and novelty. We do not claim that enterprises lack decision trees or human review, nor that the decision list itself is new. The research object is a model's design-time compliance with a stated autonomy policy that permits no AI and, when justified, deferral. The potentially consequential failure is a plausible recommendation with more operational authority than that policy permits: 75/87 direct errors across six models are upward; even the strongest direct baseline, Haiku 4.5, has eight upward errors among its 120 decisions (Table 2). The contribution is therefore a policy-conditioned measurement setting, a directional failure characterization, and a compiled extraction/decision alternative with explicit assumptions and coverage costs—not evidence of adoption or behavioral change inside organizations. We will make that positioning prominent on the first page.

2. Circularity of the policy, fictional cases, and policy-derived answers. We accept the distinction at the heart of this criticism. Because our target is compliance with a disclosed policy, policy-derived labels specify the reference behavior; they are not evidence that this policy identifies the optimal architecture for real organizations. Human assessment adds a blinded recoverability and sensitivity check, not external architectural validation: five AI research scientists judged requests without seeing the construction labels; the two who helped author some scenarios did not judge their own. Consensus matched 115/120 policy-derived labels, with two different level judgments and three cases found underspecified (Sec. 5). The upward direction remains with practitioner-consensus references (80/96 errors; Sec. 8). Nonetheless, all cases are fictional and the policy is author-defined. Independent requests and independently established organizational policies are needed before making real-world validity claims, and we will say so explicitly.

3. Could the AI-oriented instruction itself create the result? This is a valid alternative explanation. The submitted controls show that upward errors persist when a request does not mention an agent (45/55) and on D1–D4 references where either error direction is possible (32/44), but neither neutralizes the system instruction. Our post-submission reanalysis of archived strict-guide annotator outputs also finds 42 upward versus 13 downward disagreements (Table 16), though that guide still refers to autonomy levels; it is not a controlled neutral condition. We therefore plan a paired neutral-introduction ablation with a separate schema-wording arm, and matched request-level preference edits that preserve business facts. We will report attenuation or null effects as well as persistence. Until then, the supported claim is an upward-error asymmetry under the disclosed policy and evaluated prompting conditions, not an intrinsic, framing-independent model bias.

We also acknowledge the presentation concern and will move the estimand, recommended direct baseline, and limitations earlier in the paper.

hank you for motivating the targeted controls proposed in our earlier response. We have now completed the six-model framing ablation and reanalyzed the labels of annotators uninvolved in benchmark construction. We refer to the all-reviewer update for the broader results and focus here on your framing and circularity questions.

1. Controlled framing ablation. On the same 120 requests with identical policy, decoding, and output schema, replacing the solution-design introduction with a technology-neutral reviewer instruction reduces over-recommendations from 81 to 67 of 720 model-request decisions. Repeating the original prompt gives 77, providing a run-to-run comparison. The paired reduction under neutral framing is 1.9% (request-clustered 95% CI for the reduction: 0.6–3.5 points). Crucially, 67 of the 81 directional errors under neutral framing remain upward. Replacing only the term “AI autonomy level” with “solution pattern” leaves 80 over-recommendations. These controls quantify framing sensitivity while independently reproducing the directional structure under another task introduction.

2. Reference-label robustness. The three annotators who did not construct the benchmark agree with the hidden policy references on 195/216 labels (90.3%). Where two such annotators agree on a level, that level matches the reference in all 70/70 cases. Using panel-consensus labels for 117 requests, the directional result persists (84 too high versus 14 too low under the original prompt). This provides an additional, construction-independent check of the observed error pattern under the disclosed policy.

3. A concrete audit implication. Checking every D4/D5 recommendation would examine 295/720 decisions and identify 62/81 over-recommendations; for Sonnet 4.6, it would examine 47/120 and identify 8/9. The measured error distribution therefore supports a precise, targeted review rule for high-autonomy proposals.

We will incorporate the framing controls, independent-label analyses, and audit calculation into the revision, with the prompts and underlying outputs included in the supplement.

Thank you so much for your thoughtful engagement with my review.

I won't be changing my score.

Reviewer_1bM1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates a critical upstream design problem in enterprise AI deployment: how to select the appropriate AI autonomy level for a specific request under a given organizational policy. The authors propose a 6-level autonomy taxonomy (D0 to D5) and release an expert-evaluated benchmark dataset, AUTOSEL-120, along with Clarify-24 for scenarios with missing information. Experiments show that current mainstream large language models exhibit a significant "over-agentic" bias when making such decisions, tending to recommend higher-level AI solutions that exceed policy limits. To mitigate this issue, the authors propose a "Compiled Autonomy Selection (CAS)" method. This approach utilizes LLMs to extract three-valued logic facts and executes the policy via a deterministic logic compiler, thereby achieving conditionally sound architectural decisions and proactively deferring decisions (DEFER) when information is insufficient.

Critical strengths
  1. Novel and highly practical problem setting: While most current work focuses on the construction and operational planning of Agents, this paper shifts the focus upstream to explore the core issue of "whether an Agent (or even AI) needs to be built," which is crucial for enterprise compliance and cost control. Including "No AI (D0)" as an output option fills an important blind spot in the field.

  2. Solid and deep empirical findings: The paper reveals the prevalent over-agentic bias in LLMs and rigorously rules out the possibility that this conclusion is influenced by the dataset's surface format through strict ablation studies (controlling bottom-line effects, prompt rewriting, lexical classifier comparisons, etc.).

  3. Innovative neuro-symbolic decoupling design: The CAS method transforms error-prone complex reasoning into a two-step process: "LLM extraction of atomic facts" + "symbolic logic execution." This design provides a theoretical guarantee of "Conditional Soundness" under the premise of accurate known information, which is highly valuable for enterprise governance scenarios.

  4. Strict annotation and evaluation standards: The benchmark dataset underwent independent blind reviews by 5 domain experts, achieving a Krippendorff's α of 0.87. This demonstrates the high consistency and reproducibility of the policy definitions and reference labels.

Critical weaknesses
  1. Small scale of synthetic evaluation dataset: AUTOSEL-120 contains only 120 samples, and all scenarios are fictional backend management use cases generated with human/LLM assistance. The lack of lengthy, ambiguous real-world enterprise tickets may limit the generalizability and persuasiveness of the model's performance.

  2. Over-refusal at the D0 level (recall bottleneck): Although the CAS method and fine-tuning reduce unauthorized recommendation errors, this significantly sacrifices coverage. Specifically, when handling D0 (No AI) scenarios, the system acts conservatively and easily chooses DEFER (successfully recalling only 1 out of 20 D0 samples). This implies that the model has not truly grasped the applicability boundaries of "non-AI solutions."

  3. Limited policy generalization and scalability: The current CAS compiler is deeply tied to the 13 predefined three-valued fact factors. If an enterprise has compliance requirements across different dimensions, developers must manually redesign the logic tree and retrain the extractor, which lacks the flexibility to automatically align with natural language policies.

Questions for authors

1.Considering the distribution differences between synthetic data and real-world enterprise data, do you plan to expand the benchmark in the future by introducing complex requirement texts from real systems (e.g., anonymized IT tickets)?

2.Regarding the extreme conservatism and low recall rate (1/20) for D0 (No AI) scenarios, do you believe the fundamental bottleneck is the inherent cognitive bias of current LLMs (a tendency to use technology to solve everything), or is it due to the lack of certain specific constraint features within the fact extraction framework?

3.If a policy cannot be clearly deconstructed into the 13 predefined three-valued factors, how should the CAS framework be adapted in the most lightweight manner?

3 comments

Thank you for recognizing the practical relevance of deciding whether an agent is warranted. We address the three weaknesses/questions in turn.

W1/Q1 — Synthetic benchmark and real enterprise requests. We agree that 120 fictional, level-balanced scenarios do not establish real-ticket generalization. The paraphrase experiment (Table 9) changes surface registers (Slack, RFP, email, meeting notes), but does not reproduce the incompleteness, organizational constraints, and context distribution of actual solution-intake tickets. The five-person panel evaluates recoverability of labels under the given policy (nominal Krippendorff's alpha = 0.87; consensus agrees with 115/120 policy-derived references), not whether the policy would yield the best real-world architecture. We will foreground this distinction. Independently sourced enterprise requirements, with separate policy-compliance and practical-feasibility assessments by solution architects and governance reviewers, are the next validation step; we do not claim to have completed it.

W2/Q2 — Why does D0 fail? The results do not support class scarcity as the sole explanation. The certified synthetic corpus contains 547 D0 scenarios; a TF-IDF classifier trained on it has D0 recall 14/20, while the defer-aware 8B factor-extractor seeds each have 1/20 (Table 3). Their principal D0 failure is often deferral on an adequately specified D0 request, not necessarily a wrong non-abstaining class. A non-defer-aware factor extractor reaches 11/20 on one seed, so learned uncertainty behavior matters as well. We suspect the first factor, whether a process or training fix suffices without AI, is the bottleneck: unlike approval or API availability, it requires a substantive sufficiency judgment. Factor extraction may therefore relocate, rather than eliminate, the architectural reasoning. We will audit D0 decisions to separate false factor assertions, unwarranted unknown values, and compiler outcomes. This is a mechanism to test, not yet a causal result.

W3/Q3 — New policies and factor vocabularies. CAS permits zero-call recompilation when a changed decision list is expressible over the existing factors (Appendix I). A genuinely new governance requirement instead requires a new factor, an updated extraction schema and evidence checks, and evaluation of the extractor; a trained extractor may need new supervision or retraining. The policy is manually encoded as a decision list, not automatically compiled from arbitrary natural-language rules. We will make this adaptation recipe and its limits explicit rather than treating a simple no-D5 restriction as proof of general policy transfer.

Thank you again for the specific questions. Following our earlier response, we completed the promised per-case D0 audit and evaluated additional zero-shot extractors under the same factor schema. The broader public-record pilot is summarized in our all-reviewer update; here we focus on what the new analysis adds to your Q2 and Q3.

Q2 — Pinpointing the D0 decision factor. The relevant policy factor asks whether a process or training fix suffices without AI. Across the 20 D0 requests and three defer-aware 8B seeds (60 case-seeds), the extractor asserted this factor as yes in three cases; all three were compiled to D0. The archived traces also distinguish "no" from "unknown" at this precise factor, allowing us to identify the upstream evidence judgment responsible for each downstream decision. Whenever the D0 factor was correctly asserted, the compiler applied the D0 rule correctly.

Cross-model verification. Using the identical extraction prompt, Qwen3-32B with reasoning and Sonnet 4.6 each recognized the process-fix factor on 13/20 D0 requests; Haiku 4.5, Llama-3.3-70B, and gpt-oss-120B recognized it on 14/20. This demonstrates that the factor is representable in the proposed vocabulary and recoverable across model families. The audit identifies a concrete target for improving the extractor without changing the policy compiler.

Q3 — Testing policy sensitivity. We additionally ran explicit D4/D5 policy variants with frozen references. Across six models, 110/120 decisions followed a policy revision assigning model-judged writes to D4, and 114/120 followed the mirrored D5 assignment. This directly validates sensitivity to policy changes expressible through the existing decision vocabulary, consistent with the adaptation procedure described in our earlier response.

We will include the per-case D0 traces, factor-recognition comparisons, and policy-variant prompts in the supplementary material. These additions make the D0 outcome diagnostically interpretable and provide directly testable evidence for policy-grounded decisions.

I appreciate the authors' thorough rebuttal. Based on these improvements, my final recommendation is weak acceptance.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Shared response to all reviewers

We thank the reviewers for three careful and constructive reviews. Two concerns recur across the reports; we address them here and respond to reviewer-specific questions separately.

What the paper claims. We study compliance with a disclosed autonomy policy, not discovery of the best architecture for a real organization (Sec. 1). Direct accuracy across the six models is 84.2–93.3%, but 75 of their 87 errors recommend greater operational authority than the policy permits (scenario-clustered 95% CI for the upward share: [77.0%, 93.5%]); this direction predominates in five of six models (Sec. 8). We do not claim that the policy is optimal, that organizations lack architectural review procedures, or that CAS outperforms frontier direct models. Sec. 9 recommends a frontier model with an audit of D4/D5 recommendations at day zero.

Shared concern 1 (DauB, 5Uog): could task framing or request wording produce the upward pattern? We agree that the submitted controls are observational and cannot isolate the system prompt. Three additional observations help delimit the explanation, but are not causal tests:

  • Under the shared policy and task instruction, direct-model D0 recall ranges from 4/20 to 19/20, and error direction ranges from nearly symmetric (gpt-oss-120B: 10 upward, 9 downward) to entirely upward (Mistral-Small: 19 upward, 0 downward; Tables 2 and 8). This heterogeneity does not exclude a contribution from the shared framing.
  • In a post-submission descriptive analysis of the archived strict-guide annotator runs (Table 16), a different role instruction asks models to apply a published annotation guide rather than act as solution-design assistants. Of 55 disagreements, 42 are upward and 13 downward (Sonnet 4.6: 10/0; Qwen3-32B: 17/2; gpt-oss-120B: 15/11). This is not a neutral-framing intervention: the annotation task still concerns autonomy levels.
  • In a separate descriptive analysis of archived direct outputs, among 18 D0–D3 requests explicitly asking for an agent, 25/108 model decisions are above the reference (23%). Among 58 D0–D3 requests that do not mention agent-related terms, 43/348 are above it (12%) and 8/348 below it (2%). Explicit agent wording is associated with more upward errors, but upward errors also occur without it; this is not a matched comparison.

We will conduct (i) a paired neutral-framing ablation, changing the opening instruction and, in a separately reported arm, the schema/tool descriptions while keeping the substantive policy, requests, and within-model decoding conditions fixed, with the original arm rerun concurrently; and (ii) the request-level minimal pairs suggested by 5Uog, preserving all decisive task and approval facts. We will report completed results during discussion regardless of direction. If the asymmetry attenuates, we will narrow the claim to the tested framing.

Shared concern 2 (all): synthetic, single-policy, and small-scale evaluation. We agree. AUTOSEL-120 is a balanced diagnostic set (20 scenarios per level), based on fictional requests under one author-defined policy; its five human raters are all AI research scientists. The pooled upward-error estimate has a scenario-clustered interval, but its 720 model decisions are not independent observations. The ~10-percentage-point detectable difference reported in Table 13 pertains specifically to paired direct-versus-contrastive prompting, not to model rankings, seed-level comparisons, or a general power guarantee. We will state these limits more prominently. Independently sourced cases assessed by solution architects and governance stakeholders are the necessary next external-validity test; we do not have that evidence and will not suggest otherwise.

AuthorsOpenReview ↗

Additional experiments: reproducibility, policy grounding, and request interpretation

We thank the reviewers for their constructive suggestions. We completed a broad set of new controls across all six models, including prompt-framing ablations, controlled request edits, policy variants, explicit-rule tests, independent-label analyses, clarification tests, matched baselines, and a pilot grounded in public AI-use-case records.

The new evidence supports four central findings:

  1. A reproducible, directionally structured error phenomenon. In AUTOSEL-120, 75/87 errors in the submission and 81/93 errors in the new original-prompt run recommend a higher autonomy level than the disclosed policy permits. Under technology-neutral framing, 67/81 directional errors remain upward. The experiment provides a direct measurement of framing sensitivity while preserving the directional phenomenon under the tested benchmark conditions.
  2. A measurable role for stakeholder requests. Removing an explicit agent request reduces D4/D5 recommendations from 19 to 7 across 102 paired decisions. In the public-record pilot, adding a mild agent preference while holding requirements fixed raises too-high choices from 3 to 16/144 (paired increase: 9.0 percentage points; 95% CI [3.5, 16.0]). This demonstrates why separating requirements from preferred solution language matters.
  3. A clear distinction between extracting facts and applying policy. When models receive the policy-relevant fact vectors directly, five of six produce no above-rule output in 600 decisions. At the D4/D5 boundary, models follow downward and upward policy revisions in 110/120 and 114/120 decisions. These results identify natural-language-to-policy interpretation as an important target for evaluation.
  4. Independent support for reference labels and an auditable method. Annotators who did not build the benchmark match its references in 195/216 judgments (90.3%); where two such annotators agree on a level, their level matches the reference in all 70 cases. Factor extraction followed by deterministic compilation provides a traceable path from evidence to decision and a conservative option for handling uncertain inputs.

We will incorporate these results into the revision and release the prompts, reference facts, outputs, and analysis scripts. Together they strengthen the paper's contribution as a policy-grounded evaluation of autonomy recommendations and of the interpretation steps that produce them.

AuthorsOpenReview ↗

New externally grounded pilot: isolating the effect of agent-request wording

To broaden the evaluation beyond AUTOSEL-120, we constructed a six-model pilot from the 2024 U.S. federal agency inventory of AI use cases (2,133 published entries), selecting 48 back-office and service workflows. Each request retains the source entry's purpose and output description. For 36 decidable requests, we supplied the policy-relevant missing facts and froze the 13-factor references before evaluation; the other 12 preserve the source descriptions for testing clarification.

High policy alignment with neutral requests. On the 36 decidable requests (216 model decisions), the models produce 201 correct decisions under the original prompt and 203 correct on repetition.

The mild preference increases over-recommendations by 9.0% (paired 95% CI [3.5, 16.0]); the stronger wording increases them by 22.2 (CI [11.1, 35.4]). The increase appears in all six models. In 31 of the 35 stronger-wording cases, the output moves an assistant/retrieval recommendation to an agentic level.

A practical evaluation signal. On the 12 published descriptions lacking decisive facts, explicit policy deferral rules elicit clarification in 47/72 decisions, while factor extraction followed by compilation does so in 70/72. This shows how the benchmark can assess a system's ability to request decisive requirements before assigning operational autonomy.

The pilot thus adds an externally sourced, controlled demonstration of why "what work must be done" should be evaluated separately from "what technology the requester asks for". The pilot's LLM-assisted reference construction and pending independent annotation will be documented alongside the released provenance and annotation kit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.