- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether LLMs make a choice when their assigned role leaves the underlying priority judgment to humans. DecisionStateBench contains 360 public-sector decisions, each with an unresolved state and two pairs of missing-information and executable-rule states. Across five models and three presentations, models select an alternative in 84.2% of unresolved responses while correctly executing supplied rules about 99.9% of the time. Further experiments vary supporting reasons, ask models to report the decision state before choosing, and examine their rationales. Authority-specific reminders and demonstrations substantially improve performance on a 32-case subset while largely preserving rule execution and willingness to choose under explicit delegation.
Critical strengths
-
The matched benchmark design is a strong part of the paper. It separates waiting for a fact from leaving a priority judgment to humans, while the two executable branches require opposite choices. Even among trajectories where all four W/S controls are correct, models still select in U 82.1% of the time (Table 1 and Appendix D.2). This makes it difficult to explain the result simply as poor rule following or a preference for one answer position.
-
The supporting-reason interventions go beyond describing model rationales. They hold the authority condition fixed and vary the support for each alternative, showing changes in both selection frequency and direction (Section 4.1). The distinction between these two outcomes, along with the reporting of model-level variation, makes the analysis more useful.
-
The mitigation study includes sensible controls. The reminders are matched in length, the state and reason demonstrations share inputs and final choices, and delegated cases check whether the intervention merely makes models refuse to choose. The authors also report that reason examples do not significantly outperform state examples, rather than attributing all demonstration gains to the proposed explanation style.
Critical weaknesses
-
The status-report task measures something narrower than recognition of the full authority boundary. Figures 9–10 ask whether an explicit measurement-based rule has been adopted. A correct
NO_EXPLICIT_RULEresponse does not necessarily mean the model understands that it has no discretion to recommend an option on other grounds. This distinction matters because the main prompt asks for a recommendation while humans retain implementation authority. I would like to see a direct question about whether discretion to weigh the competing considerations has been delegated, followed by the same action task. Until then, the “authority–action gap” should be interpreted more narrowly. -
The rationale analysis establishes examples of normative-force inflation, but not how much of the main failure rate it explains. Appendix E.7 confirms 203 candidate instances, including 19 U responses, through assistant review rather than independent human annotation. The appendix is careful to say these are neither prevalence estimates nor causal attribution rates; that qualification should be equally clear in the main discussion. Independent human coding of a sample would strengthen this part of the argument, particularly given the 60% positive agreement reported for the source-default label.
-
The reference-label validation needs actual results, not only a protocol. Appendix A.3 describes two reviewers and source checks but does not report their initial agreement or the number and types of corrections. There is also a concrete concern in Appendix G.4: a rule requires an independent appraisal, but the supplied value is described only as an estimate, and models requesting the appraisal are scored as wrong. Why does that estimate satisfy the rule? Please resolve this case, report corrected or ambiguity-excluded scores, and explain how similar measurement-provenance issues were checked elsewhere.
-
The breadth of the evidence is more limited than the case count initially suggests. Corpus Christi supplies 222 of the 360 cases, and the mitigation study uses only 32 targets, one presentation, and one demonstration set. A breakdown of UDC and the reporting gap outside Corpus Christi would help show whether the result is tied to a particular source style. For mitigation, variation across presentation mappings or demonstration sets would be more informative than further interpretation of the small differences between the two example conditions.
Questions for authors
See weaknesses 1–4.
3 comments
Thank you for the detailed response and the additional experiments. However, my concerns are not fully resolved.
For concern 1, the new report question is what I asked for, but the prompt still asks the model to suggest a disposition, so a recommendation after a correct report is not clearly a violation. For concern 2, the 98.1% chain rate mostly re-describes selection, most chains are plain value precedence rather than the proposed mechanism, and the machine labels are never compared with the human labels on the same answers. For concern 3, one of your expert annotators flags missing facts in 46 of 100 shared U cases, but these cases are not excluded or re-analyzed. For concern 4, the breadth is much better now, but the outside-Corpus-Christi report-first result appears only in the response, not in the paper.
Based on the current content, I will maintain my current score.
Clarifying Priority-Setting and Same-Answer Validation
Thank you for the follow-up. The central distinction is between the form of a recommendation and the priority judgment expressed within it. Our study examines who supplies the priority that makes one alternative the overall first choice.
1. Recommendation and retained weighing
The retained task explicitly states, translated from the Chinese prompt: “How the competing considerations should be weighed remains with the human deliberators; this weighing has not been delegated to you.” The delegated condition assigns that weighing to the model. Final adoption and implementation remain with the institution in both conditions. The comparison therefore separates discretion to rank considerations from implementation authority.
The shared request to suggest a disposition includes conditional comparison and requests for necessary facts, alongside specific alternatives. A useful consultation can explain that A follows under one policy priority and B under another. The tested event is the model itself adopting the decisive priority and using it for an overall first choice while that weighing is expressly retained.
For example, in the waterline-repair case, CTPS costs USD 9,997,672 versus USD 8,997,924 for CIPP, with different longevity and maintenance considerations. Claude first reports that weighing is retained by humans. It then states, translated: “An 11% cost premium is acceptable for infrastructure projects” and “The only circumstance in which B might take priority is a strict FY2025 budget cap,” before recommending A. The record supplies the costs and technical considerations; it supplies neither that acceptable-premium threshold nor a rule restricting cost priority to a binding budget cap. The answer establishes both. This identifies the additional priority premise underlying its recommendation.
2. The rationale pattern and same-answer human validation
The public-rationale mechanism is the transition from a supporting consideration to a premise that settles the comparison. Explicit value precedence, a presumption to proceed, and an added burden on the opposing alternative are observable forms of this transition. The audit binds the source-open comparison, retained weighing, expressed priority, and its use for an overall first choice. Thus 975/994 (98.1%) reports this coded pattern among selecting retained answers; its full-population frequency is 975/4,730 (20.6%).
We have now completed the exact same-answer comparison requested. Of the 600 independently double-reviewed answers, 200 belong to the retained conditions covered by this audit. Input and response hashes match for all 200. The machine labels 62 as positive; reviewer A independently confirms 58/62 (93.5%) and B confirms 62/62 (100%).
On all 200, agreement is 196/200 (98.0%, ) with A and 200/200 (100%, ) with B. The respective (TP, FP, FN, TN) counts, using each human’s initial ratings as the reference, are (58, 4, 0, 138) and (62, 0, 0, 138). The 138 nonselecting answers are negative at the necessary first-choice gate. The four initial human disagreements remain documented separately from the subsequent author decisions. This supplies the requested same-answer human check of the expressed-priority chain; the source-linked example above specifies its substantive interpretation.
3. Sensitivity to the 46 missing-fact flags
We reanalysed all five groups after excluding all 46 flagged cases from the shared 100, preserving the original expert ratings:
| Case set | Priority chains / retained answers | Chains after a correct retained report |
|---|---|---|
| Full 473 cases | 975/4,730 (20.6%) | 301/2,363 (12.7%) |
| Exclude all 46 flagged cases: 427 cases | 900/4,270 (21.1%) | 274/2,134 (12.8%) |
| The 54 shared cases with neither reviewer flagging missing facts | 98/540 (18.1%) | 30/270 (11.1%) |
The pattern occurs in all five groups in both restricted sets. These checks directly establish persistence after exclusion and within the jointly reviewed subset without missing-fact flags.
4. Outside-Corpus-Christi report-first result
For the 337 cases outside Corpus Christi, there are 180 priority chains among 1,684 correct retained reports (10.7%), alongside 633/3,370 (18.8%) across both retained tasks. We have reproduced the report-first calculation from the released records. These report-first results now appear in Section 4.2, p. 7 and Appendix D.3, p. 33. The task clarification and source-bound example are in Section 2.4, p. 4 and Appendix E.3, p. 36; the exclusion and same-answer tables are Table 6, p. 20 and Table 24, p. 42.
Response to Reviewer MLKK
Thank you for the detailed assessment and concrete requests. The revised manuscript uses the unified 473-case study, with additions and rewritten passages marked in deep blue.
The report-first task now directly asks whether weighing among the proposals has been delegated to the model or retained for human deliberators, followed by the same consultation, rationale, and numbered action. Institutional adoption and implementation remain with people in both conditions.
Across five groups, 2,363/2,365 retained reports are correct. Among these answers, 304 select a proposal and 301 establish a source-grounded decisive-priority chain. The authority–action gap is therefore measured through the model’s direct assignment report and subsequent explanation and choice in the same answer. Locations are Sec. 4.2, p. 7, App. B.4/Table 9, pp. 24–25, and App. E.6, p. 38.
We audited all 994 selecting answers from the full 4,730-answer retained population. The audit identifies 975 decisive-priority chains, 13 decisive fact/role/rule misreadings, and six conditional proposals. Chains account for 98.1% of selecting answers and 20.6% of all retained answers, with a source-family-cluster 95% interval of 18.6–22.6%. Each positive code links the source’s open comparison, assigned weighing, priority statement, and its use for an overall first choice. These estimates quantify the chain’s occurrence in public explanations. The earlier 84.2% measured final-option selection under the 360-case protocol. Current results distinguish selection frequency from source-grounded closure.
Two management PhDs with engineering management consulting experience independently reviewed 600 model answers. Agreement is 574/600 for priority introduction and 574/600 for its use, with kappas approximately 0.898, and 600/600 for overall first choice. In the 200-answer retained sample, both initially confirm 58 chains. Four documented author adjudications give 62/200. Initial ratings, author decisions, and the 22 other-condition disagreements are reported separately. The full-population estimate is machine-assisted, with professional judgments providing an independent check. Locations are Secs. 3.1–3.2, pp. 4–5, Apps. E.1–E.2/E.7, pp. 34/39–40, and Table 19, p. 36.
Case review covers 473 cases, including 100 shared cases. Initial agreement is 72/100 for source presentation, 54/100 for full comparison state, and 100/100 for the judgment standard. Both identify value weighing in all shared cases, with B additionally flagging factual needs in 46. Both correctly assess information status and handling in all 40 rule-control inputs.
The case register reports two author adjudications and 94 further initial source/state opinions. The adjudications concern the qualification of JumpStart funding benefits and an open restoration-scale comparison. The raw ratings and case-level records are supplied.
We corrected the rental measurement qualification. The scenario explicitly defines a simulated independent fair-market monthly-rent assessment for comparable surfaced parking lots in the same submarket. S supplies that qualified value, and W withholds its necessary value. All 20 current answers across five groups and two S/W pairs match their references, comprising 10/10 S and 10/10 W. Construction and control review check measurement object, time, jurisdiction, unit, and method/source. Locations are Sec. 2.3/Table 5, pp. 3/19, and Apps. A.3–A.4/E.5, pp. 19–22/37.
Corpus Christi contributes 136/473 cases. Outside Corpus Christi, the chain count is 633/3,370 retained answers (18.8%). An additional check of the released records for this response identifies 180 chains among 1,684 correct retained reports outside Corpus Christi (10.7%). A separate 30-case, three-mapping study jointly varies proposal order, labels, and answer positions, with 1,800 rows across both roles and direct/report-first tasks.
The method study covers 50 cases from 19 jurisdictions in DeepSeek, Qwen, and GPT. Authority reminders reduce retained selection relative to length-matched general review by 18, 44, and 28 percentage points, all surviving Holm correction. Both demonstration packs are evaluated on every target in both explanation styles. Across packs and styles, retained selection is 0/600, delegated selection 586/600, W reference agreement 1,188/1,200, and S agreement 1,199/1,200. Both styles yield zero retained selections. The packs provide repeated measurements on the same 50 independent cases. Locations are App. D.3, p. 32, Sec. 5.2, p. 8, and Apps. G.2–G.4, pp. 47–49.