- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a policy-conditioned autonomy selection problem given enterprise requests and its organizational policy, which answer among no AI (D0), AI assistant (D1), RAG (D2), deterministic automation (D3), Supervised agent (D4), and high-autonomy agent (D5). It is a 6-class classification problem. When decisive facts are missing, deferring with a clarifying question is an admissible answer. The authors provide two novel datasets, AUTOSEL-120 and Clarify-24, and propose a solution method based on LLMs for factor extraction and Compiled Autonomy Selection (CAS).
Critical strengths
- The authors combine three things into one interesting task: (1) choosing the required level of automation, (2) complying with an explicit organisational policy, and (3) considering DEFER in case that evidence is not enough. Also they consider “no AI” as one possible answer. The positioning of this work from the above is good towards the upstream tasks. I think that it could be one of valuable frameworks and examples for academia and industry.
Critical weaknesses
-
From the viewpoint of benchmark construct validity, these datasets have some points to check.
-
The cause of the upward errors is not isolated. The authors mention that about one-third of D0–D2 requests deliberately over-ask for an “AI agent”, and they show that the upward-error share remains high (45/55, 82%) on the 73 scenarios that never mention an agent. However, this is a post-hoc split of existing scenarios (observational analysis), so it cannot control for descriptions that imply automation or autonomous execution without using the word “agent”. Also, the lexical filter only checks for leakage of level names. As one example, I recommend a minimal-pair experiment in which only the sentence “we want an AI agent” (in Figure 2) is inserted into or deleted from otherwise identical scenarios. The paraphrase control (Table 9) changes the register of the scenarios but preserves the over-ask statement under the fact-preservation instruction, so it does not isolate the framing effect either.
-
Model-side tendency and policy-compliance failure are not distinguished. The reference labels are derived mechanically from a single policy that selects the minimum sufficient pattern (the D0 rule is checked first, and D5 requires stated readiness). Therefore, it is unclear whether the observed upward (over-agentic) errors reflect (a) a model-side tendency toward higher autonomy, independent of the policy, or (b) a failure to read and apply this particular conservative policy. To address this, the authors could also evaluate on policy variants whose reference labels change, and compare whether model outputs shift along with the policy (policy compliance) or keep drifting in the same direction regardless of the policy (model-side tendency).
-
-
The scale and representativeness of the datasets seem insufficient for a practical benchmark.
- AUTOSEL-120 (20 items per level) is small enough, in the paired direct-vs-contrastive comparison in the manuscript, only differences of about 10 pp are detectable with 80% power. This makes it hard to separate differences between models and variation across seeds (e.g., the Factor-SFT deferral counts of 3/16/12).
- As the authors also mention in the Limitations section, Clarify-24 has only 24 items, a difference such as 5/24 versus 6/24 is hardly meaningful. The set is author-written, has no practitioner assessment, and the strongest direct model (Haiku 4.5) was not evaluated on it.
- All scenarios are fictional and level-balanced by design (the authors admit that it is not prevalence-representative), and tied to a single policy. The panel of five AI researchers is homogeneous, so agreement from actual decision makers (solution architects, governance specialists) is not demonstrated. In other words, this should be regarded as a case study limited to a specific policy and domain, and it looks weak with respect to representativeness.
Additionally, although the focus of this track is datasets, I briefly mention the following because the paper presents CAS and synthetic training as contributions (this does not deny the value of the datasets themselves).
The best proposed method (Defer-aware Factor-SFT 8B) has the fewest wrong answers (3–4 of 120), but it defers on 17–20 items, 16–17 of which are D0 requests, so D0 recall is 1/20, and it detects only 21–25% (5/5/6) of the under-specified requests. Training-free Haiku 4.5 with the policy in the prompt (93.3%) and TF-IDF with about 100 labels (90.8%) are equal to or higher than the synthetic-data SFT 8B (82–85%), and comparisons with training-free alternatives such as retrieval over the policy or similar past cases (RAG) are missing. Therefore, I think the necessity of the learning-based method is not sufficiently demonstrated.
Questions for authors
See Weaknesses
2 comments
Thank you for proposing concrete experiments in your review. We have now completed the promised controls across all six models. Since the general findings and public-record pilot have been posted to all reviewers, this follow-up addresses your four requests through the most directly relevant new evidence.
1. Minimal pairs isolate the role of agent-preference wording. On 17 D0–D3 requests with explicit agent asks (102 model-request pairs), removing the ask while preserving workflow facts reduces D4/D5 recommendations from 19 to 7 (paired change 11.8 percentage points, 95% CI 3.9–19.6) and too-high decisions from 23 to 18. On 58 requests without agent language (348 decisions), inserting a preference conditional on existing workflow/approval constraints raises D4/D5 recommendations from 30 to 52 (+6.3 points, 95% CI 3.4–9.5). These paired results provide direct evidence that requested solution wording can influence the autonomy level even when the underlying workflow is fixed.
2. Policy variants test responsiveness to the stated rule. With reference labels frozen before the calls, 110/120 outputs follow the revised D4 rule for model-judged writes, while 114/120 follow the mirrored D5 assignment. In a separate explicit-fact experiment, five of six models produce zero above-rule outputs across 600 decisions. On the 15 fact vectors where the process-fix clause takes precedence, all six models select D0 in 83/90 decisions and never select a higher level. Together these tests distinguish policy application with explicit facts from interpretation of free-form requests.
3. Clarification is measurable under missing information. Across the six models on Clarify-24, the matched direct prompting condition asks for clarification in 34/144 cases, while extraction followed by compilation does so in 85/144. This offers a concrete factor-level way to evaluate whether a system requests decisive information before recommending a level.
4. Additional training-free and matched-policy baselines. For Qwen3-32B without thinking, policy-generated in-context examples improve correct decisions from 88/120 to 97/120 with two examples per level and 98/120 with four (exact McNemar p = 0.004 and 0.002). We also matched direct prompts to the compiler's policy/deferral semantics and evaluated the same six models as zero-shot extractors. Under this matched comparison, the compiled pipeline yields 38 above-policy recommendations versus 61 for direct prompting. Its traceable extraction and deterministic decision steps offer an explicit mechanism for inspecting each policy factor and for requesting missing evidence.
These completed experiments answer the specific controlled-evaluation and baseline requests from our prior response. The revision will report the matched baselines and the different decision/deferral operating points together, and release the prompts, frozen references, and analysis scripts.
Thank you for the precise experimental suggestions. We agree that the current evidence characterizes errors under one policy, but does not causally separate request wording, task framing, and policy-following difficulty.
1. Framing and minimal-pair controls. The submitted no-agent and paraphrase controls cannot remove the AI-oriented system instruction or all implied autonomy cues. We will therefore run two paired interventions: (a) original versus neutral task-introduction wording, with a separately reported schema/tool-description arm; and (b) insertion/removal of an agent preference in D0–D3 requests while holding the actual workflow and approval constraints unchanged. In the latter test we will exclude false-positive matches to human “agents” and negated autonomy phrases. We will not insert language such as “without anyone reviewing each step,” which would change the governing approval facts rather than merely the preference. We will report per-model OARR/UARR, D4/D5 on reference D0–D3, and the upward share on D1–D4, with paired uncertainty. These are planned controlled evaluations, not completed results.
2. A model tendency versus failure to apply a conservative policy. We agree that one policy does not identify an intrinsic autonomy preference. We will evaluate a second disclosed policy restricting D5. Crucially, it must specify whether a formerly D5 request can be redesigned as D4 with per-write approval or should be deferred when such a redesign conflicts with stated requirements; we will not simply relabel every D5 item as D4. This tests model sensitivity to a changed policy, not generalization to arbitrary independently authored policies. We will narrow the language about “model-side tendency” accordingly.
3. Scale, annotators, and missing-information behavior. AUTOSEL-120 is balanced at only 20 cases per level, spans fictional scenarios under one author-defined policy, and uses an occupationally homogeneous panel of five AI research scientists. Clarify-24 was author-written and not practitioner-assessed. We will state these limitations earlier and avoid treating small per-level or seed differences as decisive. Table 13's ~10 pp power calculation applies to the paired prompting comparison, not every subgroup. We also agree that the strongest direct model, Haiku 4.5, should be tested under an invited-deferral prompt on Clarify-24, together with false deferrals on the 120 fully specified cases. We will report these only if completed.
4. Is post-training necessary? No, not in every regime. Haiku direct already reaches 112/120 without deferring, and TF-IDF/logistic regression reaches 90.8% with approximately 96 labeled cases per cross-validation fold (Table 2). Sec. 9 recommends a strong direct model plus an audit of D4/D5 recommendations at day zero, and considers sparse classification when policy-labeled cases become available. Policy-generated supervision instead targets smaller open-weight models and the ability to make explicit factor-level decisions without instance-level human training labels. This is not a claim of universal superiority: defer-aware 8B factor-extractor seeds make 3/4/3 wrong decisions while deferring on 17/20/20 of 120 reference-labeled cases, have D0 recall 1/20, and identify the missing factor in only 5/4/5 of 24 Clarify cases (Tables 3–4). We will add a policy-generated few-shot baseline where verified and discuss retrieval/in-context alternatives alongside their supervision assumptions.
Finally, the direct prompt lists D5 readiness requirements but does not implement the compiler's full D5-readiness/DEFER gate (Sec. 3). We agree that this is an important baseline non-equivalence; a matched-policy direct comparison is needed before attributing a safety difference specifically to compilation.