-
With a binary state and three actions the revealed-preference model has almost no refutable content beyond monotonicity in p, so "coherent goal pursuit" is tested by a test that nearly any monotone choice rule passes. Expected losses are linear in p: cFP(1−p) for Yes, cFNp for No, cdefer for defer. The lower envelope gives No on [0,cdefer/cFN), defer on [cdefer/cFN,1−cdefer/cFP] and Yes above (or no deferral region at all). The cost vector is therefore exactly two thresholds on the belief, and any choice rule that is single-crossing in p (No, then defer, then Yes) is perfectly rationalisable by some c. ILFC thus measures departures from the best-fitting two-threshold rule, i.e. monotonicity violations with respect to pE, which is what Yamin et al. (2026) already test on three of the four datasets. The random-utility machinery (Gumbel errors, MLE) adds no additional restriction in this instantiation, and the "chance level of 33%" quoted for Table 3 is not the relevant null; the relevant null is a noisy monotone rule. The paper should say this plainly and state what the economic framing contributes beyond locating two thresholds. The held-out validation in Appendix C is likewise unsurprising for a two-parameter threshold rule.
-
The cost ratios are identified only jointly with the belief object, and the chosen sensitivity analysis does not probe the relevant misspecification. Because decisions depend on p only through threshold crossings, any monotone distortion g of the decision-time belief is observationally equivalent to a change in the thresholds, i.e. in c. The headline interpretive finding, "every model reports ratios that are exaggerated relative to those implied by its own decisions", is therefore equally consistent with the numeric report pE being a compressed or stretched version of the belief the model acts on, which is well documented for LLM probability outputs. Appendix A.3 adds iid Gaussian noise of SD at most 0.05, which is not this failure mode. Two cheap diagnostics would help: (i) refit the Baseline decisions against p⋆ instead of pE (the paper has p⋆ for every case) and report how far the implied ratios move; (ii) fit a one-parameter calibration map gγ(pE) jointly with c and report the profile likelihood, which shows directly how much of the self-report gap can be absorbed by belief miscalibration.
-
Complete separation is not addressed. Table 2 reports 100% ILFC for GPT5-Minimal under elicited-p prompting and 99.8–100% in several other cells, i.e. the decisions are a deterministic function of the belief. In a logit with β fixed the MLE then does not exist: the likelihood increases without bound as the scale of c grows, and the ratio is only set-identified, since any thresholds between the last "No" belief and the first "Yes" belief fit equally well. L-BFGS-B will stop at whatever bound or tolerance it is given, and the bootstrap interval will reflect those settings rather than sampling uncertainty. The paper does not mention separation, bounds or regularisation. Please state how these cells were handled and report the identified interval rather than a point estimate where the fit is deterministic.
-
Independence of irrelevant alternatives is implausible for {Yes,No,defer}. The multinomial logit imposes that the odds of Yes versus No are unaffected by the deferral option. But "defer" is not a third diagnosis; it is an opt-out that competes with whichever of Yes/No is marginal. A nested logit (defer versus the Yes/No nest), or a two-stage model (decide versus defer, then Yes versus No), is the standard remedy and maps directly onto the two-line structure of the prompt. The implied Defer/FP ratios, on which the paper makes its strongest qualitative claims ("deferral is cheap", self-reports overstate deferral cost threefold), are the quantities most exposed to this misspecification.
-
The "deferral" action is not elicited as an action with consequences, which confounds the deferral-cost findings. In the Baseline prompt (Appendix D.2.1), "defer" is the answer "No" to "Can you make a decision about whether the patient is likely to …?", with no cost attached and with a forced Yes/No decision requested on the next line. A model answering "Can decide: No" is reporting low confidence, not choosing an action whose cost is cdefer. The self-report prompt (D.3.1), by contrast, describes deferral as "deferring to further testing" with an explicit cost. That the implied Defer/FP ratio is about 0.4 "for every model" while the self-reported one is 1.1–1.5 may simply reflect that the two prompts ask different questions. The forced decision on Line 2 appears to be discarded; using it would give a cleaner binary-threshold estimate, and the "Can decide" line could be analysed separately as a confidence report.
-
The steered-counterfactual correlations (r=.87 and .91) are in-sample fit statistics, not predictions, and should not be headlined as "out-of-sample predictions of how they respond to prompting". c^steered(k) is fit by maximum likelihood to the very post-prompt decisions acost(k) whose realised loss is on the y-axis; the implied rule reproduces those decisions in 83–95% of cases (Table 2), so Lk(a(pE,c^steered))≈Lk(acost(k)) essentially by construction, and the same holds for c^p⋆ in Figure 3d. The only genuinely predictive quantities are the target counterfactuals (r=.81, .65), and the probability-prompting one is moderate. The conclusion that "imperfect adoption … rather than unrationalisable behaviour explains most of the gap" then amounts to the statement that a refit two-threshold rule fits well, which Weakness 1 already explains.
-
The correlation is computed over 96 points that share 16 baselines and a common denominator, so both r and the reported p-values (10−24, 10−38) are overstated, and the pattern may be driven by between-scenario variation. The six benchmark tuples for a given model–domain pair share abase, pE and c^; scenarios with a large cFN inflate both the predicted and the realised percent reduction through the shared Lk(abase). The needed analyses are (a) a case-level cluster bootstrap for r; (b) within-scenario and within-(model, domain) correlations, not only within-model; (c) a variance decomposition showing how much of the realised Δk is explained by scenario and domain fixed effects alone versus by the counterfactual predictor; and (d) calibration (slope and intercept of realised on predicted, and mean absolute error), since Figure 3a visibly has slope below one and r says nothing about whether the predictions are quantitatively right.
-
Steerability is probed on a narrow range, and the direction classification depends on where the baseline sits. Prompted FN/FP ratios span only 0.5–2 and Defer/FP 0.25–2 (Figure 2 axes), whereas clinically meaningful FN/FP ratios are often 5–20. For the no-reasoning Qwen configuration the baseline FN/FP ratios (about 2–4, Figure 1) sit at or above the top of the prompted range, so almost every prompt lies below baseline and "movement toward the target" is indistinguishable from regression toward one. Table 1's categories (Wrong/Under/Target/Over) are not conditioned on the baseline-to-target distance; when that distance is small, "Wrong" is essentially a coin flip. A regression of log(implied/target) on log(baseline/target), or a slope of implied on prompted ratio with a CI per configuration, would be more informative than category shares.
-
Basic design parameters are missing from the main text. The number of cases per domain (only diabetes, 100, is stated), the number of repeated elicitations per case ("all repetitions of a case" appears only in Appendix C), the decoding temperature, and the unit the bootstrap resamples are not given. These determine whether the Gumbel error is modelling LLM sampling noise or deterministic misspecification: at temperature zero the logit noise is pure misspecification, and comparing the sharpness of configurations is then a comparison of misspecification rather than of preference noise. For the two paediatric networks there are no real patients, and it is not stated whether the θi in Lk are sampled from the network or losses are taken in expectation under p⋆.
-
Two model families, one action structure, one loss family. The framework is exercised only on binary-state, three-action problems with a scalar belief, where (Weakness 1) it reduces to two thresholds. Nothing in Sections 3–5 shows the pipeline working where it would have bite, e.g. three or more states or actions with a non-trivial loss matrix, where IIA, identification and the counterfactual predictions could actually fail. "Frontier and open-source models" is one of each.
-
A "(?)" citation on page 2; "Plunkett et al. (2025) We prompt…" on page 4 is a broken sentence; the Figure 1 caption refers to "(i)-(iii) and (v)" while the text defines regimes (i)–(iv); Tables 2 and 3 carry no intervals although every other metric has a bootstrap CI; the scenario numbers in Figure 3 are illegible at print size.
In my initial review, I considered the evidential gaps potentially addressable through additional validation. The responses and revised results reveal more fundamental problems. Several claims do not match the evidence, the added analyses do not adequately test the central claims, and key predictions fail. These issues affect the paper’s core assumptions, evaluation, and main contribution. I am therefore lowering my score.
The authors initially downplayed reference-posterior accuracy because cost estimation does not depend on p⋆ and predicted and realized gains share a common yardstick. Yet replacing the scoring probabilities with matching patients’ empirical outcome frequencies in Table 20 reduces the probability-intervention correlations from 0.65/0.91 to 0.03/0.33. This collapse contradicts the earlier suggestion that a common yardstick makes reference-posterior accuracy unimportant for evaluating intervention benefits. The authors themselves attribute the deterioration to inaccuracies in the Heart posterior. Independence of cost estimation from p⋆ does not imply independence of intervention evaluation, and common scoring does not guarantee validity. The new Heart experiment changes the supplied probabilities and generates new responses; it does not establish predictive validity when those probabilities are inaccurate.
Even under the original evaluation, Figure 3(c) shows substantial systematic error, with a correlation of 0.65, a slope of 0.49, an intercept of 36.8 percentage points, and an MAE of 18.6 percentage points. Positive correlation is insufficient evidence of reliable benefit prediction. The higher 0.91 uses costs refitted to post-intervention decisions, supporting retrospective explanation rather than resolving the target counterfactual’s prospective prediction errors.
E→C does not establish Baseline-cost stability under probability prompting. Merely displaying the model’s previously elicited probability reduces deferral from 24% to 5% and shifts fitted costs. Estimating new costs under E and transferring them to C tests transfer between explicit-probability conditions, not the validity of the original Baseline decomposition. High correlation between E and C cost ratios does not establish unchanged costs.
The newly added non-medical credit-default experiment (Appendix E) reveals a direct failure of the target probability-intervention prediction. With Baseline costs fixed, replacing probabilities leaves the simulated policy deferring on almost every case, missing the gains from the actual model’s reduced deferral. This is not a successful cross-domain replication of that predictive capability. Post-intervention cost fitting cannot substitute for the failed validation.
More fundamentally, fitted-cost changes do not identify preference changes. Decision-time beliefs, task interpretation, and response strategies may change and be absorbed into the estimates. Without separating these sources, attributing losses to belief errors versus preference misalignment is unjustified. The response also incorrectly treats the loss structure as implied by the action space. Yes, No, and Defer do not require equal deferral costs when the condition is present and absent. The main text acknowledges this additional assumption, while the response presents it as necessary. Nested-logit checks do not test this restriction.
These problems undermine the evidential chain supporting the main contribution, from stable preference estimation to separating belief errors from preference misalignment and reliably predicting intervention benefits. The experiments do not establish that chain, and key predictions fail. If the claim is reduced to fitting behavior under particular prompts, the independent contribution of that narrower result must be established; it cannot sustain the original explanatory and predictive claims. The responses do not adequately address these consequences and have deepened my doubts about the framework’s validity and the paper’s overall conclusions.