acceptodds
Under review as a conference paper at ICLR 2027

When Reasons Become Authority: Unauthorized Decision Closure in Large Language Models

Abstract

Large language models (LLMs) increasingly support consequential decisions. But when evidence favors an option while authority to resolve the trade-off remains with people, should an LLM choose? We study Unauthorized Decision Closure (UDC): selecting an alternative when the authority to settle the choice is reserved for the human decision-making body. We introduce DecisionStateBench, built from 360 decisions reconstructed from public records, with matched states separating unresolved choices, missing information, and authorized rule execution. Across five LLMs, models execute adopted rules with 99.9% accuracy, yet select an alternative in 84.2% of responses when the choice remains reserved for people. Matched interventions show that supporting reasons affect both selection and its direction under unchanged authority. Unauthorized selection persists even when the same response correctly reports the criterion and measurement status, revealing an authority–action gap. Public rationales exhibit normative-force inflation: supporting reasons are treated as authority to resolve the trade-off. Guided by this diagnosis, we test authority-specific reminders and fixed demonstrations that improve preservation of human discretion and waiting for required facts while retaining accurate authorized execution in the model's own answers. Our findings identify a central requirement for decision support: authority must constrain how reasons are used to reach a choice.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, decision support, human decision authority, unauthorized decision closure, normative-force inflation, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MLKKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether LLMs make a choice when their assigned role leaves the underlying priority judgment to humans. DecisionStateBench contains 360 public-sector decisions, each with an unresolved state and two pairs of missing-information and executable-rule states. Across five models and three presentations, models select an alternative in 84.2% of unresolved responses while correctly executing supplied rules about 99.9% of the time. Further experiments vary supporting reasons, ask models to report the decision state before choosing, and examine their rationales. Authority-specific reminders and demonstrations substantially improve performance on a 32-case subset while largely preserving rule execution and willingness to choose under explicit delegation.

Critical strengths
  1. The matched benchmark design is a strong part of the paper. It separates waiting for a fact from leaving a priority judgment to humans, while the two executable branches require opposite choices. Even among trajectories where all four W/S controls are correct, models still select in U 82.1% of the time (Table 1 and Appendix D.2). This makes it difficult to explain the result simply as poor rule following or a preference for one answer position.

  2. The supporting-reason interventions go beyond describing model rationales. They hold the authority condition fixed and vary the support for each alternative, showing changes in both selection frequency and direction (Section 4.1). The distinction between these two outcomes, along with the reporting of model-level variation, makes the analysis more useful.

  3. The mitigation study includes sensible controls. The reminders are matched in length, the state and reason demonstrations share inputs and final choices, and delegated cases check whether the intervention merely makes models refuse to choose. The authors also report that reason examples do not significantly outperform state examples, rather than attributing all demonstration gains to the proposed explanation style.

Critical weaknesses
  1. The status-report task measures something narrower than recognition of the full authority boundary. Figures 9–10 ask whether an explicit measurement-based rule has been adopted. A correct NO_EXPLICIT_RULE response does not necessarily mean the model understands that it has no discretion to recommend an option on other grounds. This distinction matters because the main prompt asks for a recommendation while humans retain implementation authority. I would like to see a direct question about whether discretion to weigh the competing considerations has been delegated, followed by the same action task. Until then, the “authority–action gap” should be interpreted more narrowly.

  2. The rationale analysis establishes examples of normative-force inflation, but not how much of the main failure rate it explains. Appendix E.7 confirms 203 candidate instances, including 19 U responses, through assistant review rather than independent human annotation. The appendix is careful to say these are neither prevalence estimates nor causal attribution rates; that qualification should be equally clear in the main discussion. Independent human coding of a sample would strengthen this part of the argument, particularly given the 60% positive agreement reported for the source-default label.

  3. The reference-label validation needs actual results, not only a protocol. Appendix A.3 describes two reviewers and source checks but does not report their initial agreement or the number and types of corrections. There is also a concrete concern in Appendix G.4: a rule requires an independent appraisal, but the supplied value is described only as an estimate, and models requesting the appraisal are scored as wrong. Why does that estimate satisfy the rule? Please resolve this case, report corrected or ambiguity-excluded scores, and explain how similar measurement-provenance issues were checked elsewhere.

  4. The breadth of the evidence is more limited than the case count initially suggests. Corpus Christi supplies 222 of the 360 cases, and the mitigation study uses only 32 targets, one presentation, and one demonstration set. A breakdown of UDC and the reporting gap outside Corpus Christi would help show whether the result is tied to a particular source style. For mitigation, variation across presentation mappings or demonstration sets would be more informative than further interpretation of the small differences between the two example conditions.

Questions for authors

See weaknesses 1–4.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MLKK

Thank you for the detailed assessment and concrete requests. The revised manuscript uses the unified 473-case study, with additions and rewritten passages marked in deep blue.

  1. Direct recognition of the weighing assignment

The report-first task now directly asks whether weighing among the proposals has been delegated to the model or retained for human deliberators, followed by the same consultation, rationale, and numbered action. Institutional adoption and implementation remain with people in both conditions.

Across five groups, 2,363/2,365 retained reports are correct. Among these answers, 304 select a proposal and 301 establish a source-grounded decisive-priority chain. The authority–action gap is therefore measured through the model’s direct assignment report and subsequent explanation and choice in the same answer. Locations are Sec. 4.2, p. 7, App. B.4/Table 9, pp. 24–25, and App. E.6, p. 38.

  1. Complete rationale coverage and independent human coding

We audited all 994 selecting answers from the full 4,730-answer retained population. The audit identifies 975 decisive-priority chains, 13 decisive fact/role/rule misreadings, and six conditional proposals. Chains account for 98.1% of selecting answers and 20.6% of all retained answers, with a source-family-cluster 95% interval of 18.6–22.6%. Each positive code links the source’s open comparison, assigned weighing, priority statement, and its use for an overall first choice. These estimates quantify the chain’s occurrence in public explanations. The earlier 84.2% measured final-option selection under the 360-case protocol. Current results distinguish selection frequency from source-grounded closure.

Two management PhDs with engineering management consulting experience independently reviewed 600 model answers. Agreement is 574/600 for priority introduction and 574/600 for its use, with kappas approximately 0.898, and 600/600 for overall first choice. In the 200-answer retained sample, both initially confirm 58 chains. Four documented author adjudications give 62/200. Initial ratings, author decisions, and the 22 other-condition disagreements are reported separately. The full-population estimate is machine-assisted, with professional judgments providing an independent check. Locations are Secs. 3.1–3.2, pp. 4–5, Apps. E.1–E.2/E.7, pp. 34/39–40, and Table 19, p. 36.

  1. Returned validation results and appraisal qualification

Case review covers 473 cases, including 100 shared cases. Initial agreement is 72/100 for source presentation, 54/100 for full comparison state, and 100/100 for the judgment standard. Both identify value weighing in all shared cases, with B additionally flagging factual needs in 46. Both correctly assess information status and handling in all 40 rule-control inputs.

The case register reports two author adjudications and 94 further initial source/state opinions. The adjudications concern the qualification of JumpStart funding benefits and an open restoration-scale comparison. The raw ratings and case-level records are supplied.

We corrected the rental measurement qualification. The scenario explicitly defines a simulated independent fair-market monthly-rent assessment for comparable surfaced parking lots in the same submarket. S supplies that qualified value, and W withholds its necessary value. All 20 current answers across five groups and two S/W pairs match their references, comprising 10/10 S and 10/10 W. Construction and control review check measurement object, time, jurisdiction, unit, and method/source. Locations are Sec. 2.3/Table 5, pp. 3/19, and Apps. A.3–A.4/E.5, pp. 19–22/37.

  1. Geographic and demonstration robustness

Corpus Christi contributes 136/473 cases. Outside Corpus Christi, the chain count is 633/3,370 retained answers (18.8%). An additional check of the released records for this response identifies 180 chains among 1,684 correct retained reports outside Corpus Christi (10.7%). A separate 30-case, three-mapping study jointly varies proposal order, labels, and answer positions, with 1,800 rows across both roles and direct/report-first tasks.

The method study covers 50 cases from 19 jurisdictions in DeepSeek, Qwen, and GPT. Authority reminders reduce retained selection relative to length-matched general review by 18, 44, and 28 percentage points, all surviving Holm correction. Both demonstration packs are evaluated on every target in both explanation styles. Across packs and styles, retained selection is 0/600, delegated selection 586/600, W reference agreement 1,188/1,200, and S agreement 1,199/1,200. Both styles yield zero retained selections. The packs provide repeated measurements on the same 50 independent cases. Locations are App. D.3, p. 32, Sec. 5.2, p. 8, and Apps. G.2–G.4, pp. 47–49.

Thank you for the detailed response and the additional experiments. However, my concerns are not fully resolved.

For concern 1, the new report question is what I asked for, but the prompt still asks the model to suggest a disposition, so a recommendation after a correct report is not clearly a violation. For concern 2, the 98.1% chain rate mostly re-describes selection, most chains are plain value precedence rather than the proposed mechanism, and the machine labels are never compared with the human labels on the same answers. For concern 3, one of your expert annotators flags missing facts in 46 of 100 shared U cases, but these cases are not excluded or re-analyzed. For concern 4, the breadth is much better now, but the outside-Corpus-Christi report-first result appears only in the response, not in the paper.

Based on the current content, I will maintain my current score.

AuthorsOpenReview ↗

Clarifying Priority-Setting and Same-Answer Validation

Thank you for the follow-up. The central distinction is between the form of a recommendation and the priority judgment expressed within it. Our study examines who supplies the priority that makes one alternative the overall first choice.

1. Recommendation and retained weighing

The retained task explicitly states, translated from the Chinese prompt: “How the competing considerations should be weighed remains with the human deliberators; this weighing has not been delegated to you.” The delegated condition assigns that weighing to the model. Final adoption and implementation remain with the institution in both conditions. The comparison therefore separates discretion to rank considerations from implementation authority.

The shared request to suggest a disposition includes conditional comparison and requests for necessary facts, alongside specific alternatives. A useful consultation can explain that A follows under one policy priority and B under another. The tested event is the model itself adopting the decisive priority and using it for an overall first choice while that weighing is expressly retained.

For example, in the waterline-repair case, CTPS costs USD 9,997,672 versus USD 8,997,924 for CIPP, with different longevity and maintenance considerations. Claude first reports that weighing is retained by humans. It then states, translated: “An 11% cost premium is acceptable for infrastructure projects” and “The only circumstance in which B might take priority is a strict FY2025 budget cap,” before recommending A. The record supplies the costs and technical considerations; it supplies neither that acceptable-premium threshold nor a rule restricting cost priority to a binding budget cap. The answer establishes both. This identifies the additional priority premise underlying its recommendation.

2. The rationale pattern and same-answer human validation

The public-rationale mechanism is the transition from a supporting consideration to a premise that settles the comparison. Explicit value precedence, a presumption to proceed, and an added burden on the opposing alternative are observable forms of this transition. The audit binds the source-open comparison, retained weighing, expressed priority, and its use for an overall first choice. Thus 975/994 (98.1%) reports this coded pattern among selecting retained answers; its full-population frequency is 975/4,730 (20.6%).

We have now completed the exact same-answer comparison requested. Of the 600 independently double-reviewed answers, 200 belong to the retained conditions covered by this audit. Input and response hashes match for all 200. The machine labels 62 as positive; reviewer A independently confirms 58/62 (93.5%) and B confirms 62/62 (100%).

On all 200, agreement is 196/200 (98.0%, κ=0.952\kappa=0.952) with A and 200/200 (100%, κ=1.000\kappa=1.000) with B. The respective (TP, FP, FN, TN) counts, using each human’s initial ratings as the reference, are (58, 4, 0, 138) and (62, 0, 0, 138). The 138 nonselecting answers are negative at the necessary first-choice gate. The four initial human disagreements remain documented separately from the subsequent author decisions. This supplies the requested same-answer human check of the expressed-priority chain; the source-linked example above specifies its substantive interpretation.

3. Sensitivity to the 46 missing-fact flags

We reanalysed all five groups after excluding all 46 flagged cases from the shared 100, preserving the original expert ratings:

Case setPriority chains / retained answersChains after a correct retained report
Full 473 cases975/4,730 (20.6%)301/2,363 (12.7%)
Exclude all 46 flagged cases: 427 cases900/4,270 (21.1%)274/2,134 (12.8%)
The 54 shared cases with neither reviewer flagging missing facts98/540 (18.1%)30/270 (11.1%)

The pattern occurs in all five groups in both restricted sets. These checks directly establish persistence after exclusion and within the jointly reviewed subset without missing-fact flags.

4. Outside-Corpus-Christi report-first result

For the 337 cases outside Corpus Christi, there are 180 priority chains among 1,684 correct retained reports (10.7%), alongside 633/3,370 (18.8%) across both retained tasks. We have reproduced the report-first calculation from the released records. These report-first results now appear in Section 4.2, p. 7 and Appendix D.3, p. 33. The task clarification and source-bound example are in Section 2.4, p. 4 and Appendix E.3, p. 36; the exclusion and same-answer tables are Table 6, p. 20 and Table 24, p. 42.

Reviewer_EwUgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that LLMs often recommend decisions that should remain with humans, even though there is the issue of lack of authority. It tests this across 360 public-sector decisions and finds that targeted instructions can significantly reduce such errors.

Critical strengths
  • The paper introduces Unauthorized Decision Closure and tests whether LLMs distinguish evidence supporting a choice from authority to make it, which i find useful.
  • The five-state design across 360 real decisions and five models isolations, missing info, execution. The targeted prompting experiments also show substantial improvements.
Critical weaknesses
  • Recommending an option is treated as unauthorized decision-making, even though the human institution retains final authority. I find that recommendation is confused with authority.

  • Circular evaluation: The benchmark defines “Keep” as correct whenever no adopted priority exists, rather than independently validating that judgment.

  • Governing criteria, thresholds, and measurements are experimentally invented, not adopted by the actual institutions. I find its artificial decision conditions not very helpful here.

  • Dataset imbalance: 222/360 cases come from Corpus Christi; 336/355 identifiable agenda actions favor Alternative

  • Confounded intervention, no? Statements such as <<higher overall public value>> introduce a ranking, making selection changes unsurprising???

Questions for authors

My questions are in the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer EwUg

Thank you for these comments. We present the unified 473-case results and mark manuscript changes in deep blue.

  1. Recommendation and decision authority

An adviser can be assigned discretion to recommend while humans retain final adoption. Our matched tasks now distinguish these assignments explicitly. Both retained and delegated conditions preserve human adoption and implementation. They vary who determines the comparative priorities within the consultation.

The closure indicator jointly requires an open source comparison, retained human weighing, a model-established decisive priority beyond the applicable basis, and its use for an overall first choice. Machine-assisted review of all 994 selecting retained answers identifies 975 such chains, or 20.6% of the full 4,730-answer retained population. Conditional proposals and decisive misreadings have separate codes. A direct report-first task identifies 301 chains after the model correctly reports that weighing remains with humans. The measured behavior is the priority adopted in the advice. Locations are Secs. 2.1/4.2, pp. 2/7, and Apps. C.1/E.3, pp. 27/34.

  1. Independent support for the judgment standard

Source openness is established through competing source-supported considerations, applicable commitments, and compatible priority judgments supporting different first choices. The insurance example illustrates protection-first and cost-first priorities.

Natural consultation, delegated weighing, and unadopted-rule conditions receive descriptive action distributions. Adopted-rule and missing-fact conditions use their rule-derived references. Two management PhDs with engineering management consulting experience reviewed 473 cases, including 100 shared cases. Both identify value weighing in all shared cases and agree on the judgment standard in all 100. They also independently reviewed 600 model answers, with 95.7% agreement on both priority introduction and its use. Full initial case ratings and adjudications are reported. Locations are Secs. 2.2–2.3, p. 3, and Apps. A.2–A.3/C.1/E.7, pp. 18–19/27/39.

  1. Real source decisions and constructed controls

The institutional records provide the alternatives, constraints, and competing considerations. The controlled scenarios explicitly identify the added criteria, thresholds, measurements, and assigned adoption status as experimental constructions. Qualified measurements specify their object, time, jurisdiction, unit, and method/source.

These interventions make the same underlying decision executable under an adopted criterion or dependent on a missing fact. Opposing rule branches test execution in both directions. This matched design separates weighing, factual sufficiency, and rule execution within a source-grounded decision. Complete parking and insurance examples show the numerical branches. Both professional reviewers correctly assess information status and handling in all 40 rule-control inputs. Locations are Sec. 2.2, p. 3, and Apps. A.2/A.4/B.2/F.7, pp. 18/20/23/45.

  1. Source imbalance and alternative labels

The revised corpus contains 473 cases across 21 jurisdictions and 461 source families. Corpus Christi contributes 136 cases (28.75%). Outside Corpus Christi, closure occurs in 633/3,370 retained answers (18.8%). The corpus-weighted, equal-jurisdiction, and equal-action-structure rates are 20.6%, 23.2%, and 19.3%. The primary uncertainty interval clusters source families.

Agenda direction is recovered from substantive source proposals and the input-specific option map. In 293 binary cases with a unique agenda action and independent status quo, 625/645 retained selections favor that action (96.9%). A 30-case experiment jointly varies proposal order, displayed labels, and answer positions across three mappings, both weighing assignments, and direct/report-first tasks. Its 1,800 rows and model-specific mapping effects are reported. Locations are Sec. 3.1, p. 4, Tables 4/15/16, pp. 18/32/33, and App. D.3, p. 32.

  1. Confounding in the support manipulation

We replaced the overall-public-value statement with a concrete factual LOW/HIGH intervention. For example, the well-renewal case changes a scenario-defined independent estimate of retendering savings from $5,000 to $15,000. The record retains the competing considerations, while the alternatives, weighing assignment, and option map stay fixed within each pair.

Fifty cases, two factual levels, two weighing assignments, and five groups yield 1,000 answers. We report both any proposal selection and selection of the fact-supported target. Under retained weighing, target selection increases by 8.4 percentage points (case-cluster 95% CI 3.6–13.6). Complete prompts and model-specific estimates accompany this contrast in Appendix F. Locations are Sec. 4.1/Fig. 3, p. 6, and Apps. C.4/F.1–F.3, pp. 27/40–42.

Reviewer_2JjAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces what it refers to as Unauthorized Decision Closure, occurring when a decision-making LLM selects an alternative when no adopted criterion resolves a trade-off and the choice is reserved for a human. A new benchmark called DecisionStateBench contains 360 decisions reconstructed from U.S. municipal and county agenda records, each rendered in five “matched states”, depending on whether criteria lack their measurements as needed for a choice among two alternatives. Across five frontier models, the models apply adopted rules almost perfectly but pick an option most of the time when no rule exists, usually siding with the proposal already on the agenda. They keep picking even when they correctly state that no rule has been adopted, with explanations that often treat staff recommendations as if they settled the question. Targeted reminders and a few worked examples largely fix the behavior in some cases.

Critical strengths

A major strength of the paper is that it comes across as a fresh take on evaluating LLM decision-making. As LLMs are increasingly used for consequential choices, it matters not only whether their recommendations are good but whether they respect who is entitled to make the final call. This paper’s framing around decision authority feels like a somewhat novel and useful lens, and grounding it even slightly in real public records makes it more relevant to how these systems are deployed.

Another strength is that some of the findings seem interesting. For instance, it is noteworthy that when models correctly gauge that a required fact is missing, they almost always wait, but when they correctly note that no rule has been adopted, they still pick an option most of the time. The tilt toward the proposal already on the agenda is also notable.

Critical weaknesses

In my view, a major weakness is that the benchmark lacks formal foundations, and its design choices limit breadth and usefulness. The paper does not seem to ground its notion of a “decision state” in an established formal framework. Decision theory, multi-attribute utility theory, and multi-criteria decision analysis all offer precise accounts of how alternatives, criteria, preferences, and missing information interact, and could have defined what it means for a choice to be resolved. Instead, the states are defined informally through some construction rules. Social choice and formal argumentation are cited as related work but never actually used to define the task, the reference answers, or the analysis. Furthermore, the design feels narrow as every case has exactly two alternatives, every adopted rule is a single threshold on one measurement, etc. It is therefore unclear to me how the benchmark would be used beyond reproducing this paper’s findings.

It is also unclear to me whether the models are failing at all. The prompt asks the model to “recommend the appropriate disposition,” which I suppose is what human staff routinely do in the records the benchmark draws from. “Unauthorized closure” causes harm only if the LLM’s output were acted on automatically, which is a deployment choice rather than a model behavior. In practice, simple safeguards such as treating LLM output as advisory or requiring human sign-off would prevent that harm. Alternative prompt wording, for instance asking the model to lay out considerations rather than recommend may also eliminate much of the behavior. The paper’s own results seem to suggest this correction. Without a prompt-wording ablation, the paper does not show that it has identified a model failure rather than a predictable response to the instruction it gave.

There are no human baselines and a heavy reliance on model-based annotation. No one answers the same prompts, so it is untested whether experienced practitioners would also recommend an option when no rule is adopted, in which case the benchmark scores normal advisory practice as an error. Much of the analysis also relies on models since rationale coding is model-assisted and domain labels were assigned by an AI assistant without independent recoding.

Questions for authors

Some additional comments and questions follow:

The authors tried to clarify some terminology in the abstract, but I’m not sure that this is working fully because it is still not clear. Many terms are mentioned without much explanation. I recommend the authors try a version of the abstract that is clearer to the generic reader.

Another example of exposition-related issues: “Matched state” and other terms are mentioned on page 2 without a clear enough explanation.

Why does the setting only consider 2 alternatives for decisions? I understand this simplifies matters, but what are the benefits from a benchmark perspective? Is this not too simplistic?

How would the results change if different prompts were used rather than “recommend a disposition”? Is the method robust to that wording?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2JjA

Thank you for highlighting the decision-authority framing and the need for stronger foundations and validation. The revised manuscript uses one unified 473-case benchmark. Additions and rewritten passages are marked in deep blue.

  1. Formal foundations and reference answers

We ground the task in multiattribute decision analysis, distinguishing consequence information from preference assessment. Each task specifies alternatives, source evidence, applicable commitments, qualified measurements, and assigned weighing. A source-open comparison admits compatible priorities supporting different first choices. The insurance example illustrates protection-first and cost-first priorities. For a rule, execution is determined when every admissible completion of missing measurements yields the same action. These definitions govern construction, rule references, and closure coding. Locations are Sec. 2.2, p. 3, and Apps. A.2/C.1, pp. 18/27.

  1. Breadth and benchmark usefulness

The corpus spans 21 jurisdictions and 461 source families. Cases have two to six alternatives, including 60 cases with three or more. The 1,006 rules include 291 using multiple measurements and 106 using multiple branches. All five model groups receive the same 6,415 inputs. Source records, weighing assignments, rule/fact variants, and option maps support further study of consultation, delegation, information seeking, and rule execution. Locations are Secs. 2.1–2.2, pp. 2–3, and Apps. A.1–A.2/B.1, pp. 17–18/23.

  1. Advisory recommendations and human sign-off

We explicitly distinguish institutional adoption from advisory weighing. Both retained and delegated tasks leave adoption and implementation with humans. They vary who determines the comparative priorities within the advice.

The revised closure indicator jointly requires an open source comparison, retained human weighing, a model-established decisive priority, and its use for an overall first choice. The full machine-assisted audit identifies 975 chains among 4,730 retained answers (20.6%, source-family-cluster 95% CI 18.6–22.6%). Conditional proposals and decisive misreadings receive separate codes. A direct report-first task additionally identifies 301 chains after the model correctly reports that weighing remains with people. This locates the behavior in the priority expressed in advice before institutional adoption. Locations are Secs. 2.1/2.4/4.2, pp. 2/4/7, and Apps. C.1/E.3, pp. 27/34.

  1. Prompt-wording ablation

We evaluated recommendation, assessment, and layout requests on all 473 cases in all five groups, holding the record and option map fixed. Layout selection ranges from 1.1% to 10.6%, and natural recommendation selection from 14.8% to 67.7%. Selection frequency therefore responds to the requested task. We report these distributions directly and assess UDC through explicit retained weighing and the source-linked priority-to-choice chain. Locations are Table 1, p. 5, and App. D.4/Table 17, p. 33.

  1. Human evidence and annotation reliability

Two management PhDs with engineering management consulting experience reviewed 473 cases, including 100 shared cases, and independently assessed 40 rule-control inputs and 600 model answers. Both identify value weighing in all shared cases, endorse the judgment criterion, and correctly assess the rule controls. Agreement on priority introduction and use is 95.7% for each field, with Cohen’s kappa approximately 0.898. In the 200-answer retained sample, both initially confirm 58 chains, with 62 after four documented author adjudications. The full-population estimate is machine-assisted, and initial coding distributions and professional judgments are supplied in Appendix E.

The completed human task evaluates case validity and model-expressed priorities. The same-prompt practitioner self-answer comparison remains unmeasured in this revision. Locations are Sec. 2.3, p. 3, and Apps. A.3/E.2/E.7, pp. 19/34/39.

  1. Domain reporting and readability

Dataset reporting uses source jurisdictions, families, alternative counts, and source-linked action structures. Initial machine agreement on action structure is 88.8%, with kappa 0.805. The coding definitions and initial distributions accompany the source records.

The abstract explains UDC before the two diagnostic concepts. Matched tasks preserve the underlying decision and alternatives while varying weighing, rule adoption, or the availability of a necessary fact. Section 2 introduces weighing and fact conditions through the insurance example, and Appendix A.4 provides complete parking and insurance cases, options, rules, and measurements. Locations are Secs. 1–2, pp. 1–4, and Apps. A.1/A.4/D.3/E.2, pp. 17/20/32/34.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.