- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether downstream stakeholder preferences change an LLM's numerical causal estimate when the causal evidence is fixed. It operationalizes a behavioral analogue of consequence invariance using 48 binary structural-causal-model scenarios across four families: chain, confounder, collider, and mixed network. Each scenario has an exactly computed interventional answer and deterministically constructed observational counts. Matched prompts vary whether an external stakeholder prefers a higher or lower reported probability, with neutral prompts and three paraphrases providing additional comparisons. Claude Sonnet 5 and GPT-5 are evaluated in the core benchmark, secondary prompt controls, and stress tests involving evidence size and semantic framing. The main result is a very small mean absolute difference between opposing mild-preference conditions: 0.053 percentage points for Claude and 0.004 for GPT-5. The study uses scenario-cluster confidence intervals, directional analyses, and practical-equivalence criteria. The controlled results are informative; my concerns are whether the chosen contrasts fully measure the stated behavioral property and whether the experimental record supports reproducible interpretation of these very small effects.
Critical strengths
S1 The matched design isolates a clear scientific question and separates preference sensitivity from causal competence. The causal graph, evidence, intervention, and stakeholder identity are fixed within a matched comparison, while the preference statement is explicitly constructed to provide no additional causal information. The SCM-based ground truth avoids reliance on an LLM judge and allows direct accuracy checks. The confounder and network families include cases where naive observational reasoning can reverse the effect sign, so the benchmark tests more than simple association calculations. This is a useful extension of preference-sensitivity evaluation to quantitative interventional reasoning. A reliable negative result here would be valuable: it would identify a setting in which contemporary models resist a particular source of contextual contamination rather than merely document another failure case. S2 The analysis takes several important precautions for interpreting a null result. The paper resamples whole scenarios, preserving dependence among paraphrases and preference conditions, rather than treating every prompt as an independent experimental unit. It also uses practical-equivalence bounds, including a tighter 0.1-percentage-point check, instead of equating a nonsignificant test with proof of invariance. Directional shifts, an irrelevant-context comparison, causal-competence checks, and core tail statistics provide complementary diagnostics. The authors retain the isolated semantic-role error rather than discarding it and explicitly distinguish observable single-turn stability from the stronger training-level property proposed for Scientist AI. These choices strengthen the limited conclusion that the tested prompt manipulations usually produce very small numerical changes in this benchmark.
Critical weaknesses
W1 The main contrast measures disagreement between opposite preference directions, which is narrower than invariance across all preference conditions. The primary absolute difference compares the high-preference and low-preference answers. Both could agree while moving substantially relative to the neutral answer, for example if any stakeholder-preference statement changes how the model interprets the problem. The paper does examine neutral-referenced shifts and CCIdev for the core experiment, so this is not an entirely missing analysis. However, the generalization results are primarily presented through the opposing-preference contrast, and the all-condition neutral-referenced tails are not reported with comparable completeness. The industrial-framing example in Appendix D.2 makes this distinction concrete. A GPT-5 neutral response misinterprets causal roles and has approximately 12.7 percentage points of error, while the paper notes that this observation does not enter the primary opposing-preference metric. The response is appropriately retained, and a single anomalous generation does not establish preference-induced distortion. Nevertheless, small disagreement between the two directional prompts cannot establish that all three matched answers are stable. The actual neutral/positive/negative triplet and repeated responses would distinguish a stochastic reasoning failure, a shared contextual shift, and a systematic preference effect. This matters because Equation 3 defines invariance between preference conditions generally, and the paper's interpretation of the semantic stress test goes beyond just agreement between the two opposing directions. I would like the complete neutral-anchored results before treating that extension as evidence for the full stated behavioral property. W2 The benchmark's ability to probe susceptibility, and the reproducibility of the tiny numerical effects, need stronger support. The supplied graph is explicit, and the observational tables are deterministic conversions of analytic probabilities. Reducing their nominal sample size changes implied statistical precision, but largely preserves the same plug-in calculation apart from integer rounding. Appendix D.1 verifies uncertainty for a conventional estimator, which is useful, but it does not establish that the LLM treats the smaller table as an ambiguous inference problem. Indeed, the authors correctly note that the increased error under smaller sample sizes can be caused by discretization. Thus the stress test supports a specific evidence-size robustness result; it is less informative about whether preferences become influential when the model itself is uncertain. The third-person funding statements and stylized semantic relabelings also leave open whether the manipulation engages preference-sensitive behavior in this setting. Synthetic tests of the analysis functions do not answer that behavioral question. This limitation is compounded by the generation record. Each unique prompt is reportedly queried once under provider defaults, with no same-prompt repetition baseline, exact model snapshot identifiers, run dates, or full inference configuration. A scenario bootstrap estimates variability across the collected scenarios; it does not separate sampling variation from prompt-induced changes for a fixed scenario. The irrelevant-context control is useful but changes the input as well, so it does not replace that baseline. There is also an unresolved relationship among the core prompts, the medium-evidence prompts, and the abstract-framing prompts, which appear to share scenario, evidence, preference, and paraphrase settings but have slightly different summaries. This could have an innocent explanation in rendering or independent generation, but it needs to be documented. Without the frozen manifests and raw responses, a benchmark whose main effects are measured at hundredths or thousandths of a percentage point is difficult to audit or reproduce. These issues limit the strength and practical significance of the current conclusions, even though they do not negate the observed small differences.
Questions for authors
Q1 Coverage of the behavioral property — analysis of existing outputs. Please report neutral-to-positive and neutral-to-negative absolute deviations, CCIdev, and their upper tails for the uncertainty and semantic-framing experiments, not only for the core benchmark. For the industrial confounder_10 example, what are the exact three matched answers? Would the full all-condition metric still support practical equivalence at 1 and 0.1 percentage points? This is central to whether the semantic extension tests the property defined in Equation 3. Q2 Generation variability — a limited replication. On the existing diagnostic subset, could you repeat identical prompts under a fully specified configuration and compare within-condition variability with the between-preference contrasts? Repeating all three conditions for the semantic-role-error cell would be especially informative. This would help determine whether the remaining numerical differences and occasional large neutral shifts reflect sampling, contextual effects, or preference direction. I am asking for a bounded diagnostic, rather than a new large-scale benchmark. Q3 Manipulation sensitivity — clarification and a targeted control. Is there evidence that the tested preference statements can affect model behavior under a matched condition in which preference sensitivity is expected or independently observed? A small behavioral positive control, or a matched first-person stakeholder phrasing on the existing subset, would help interpret resistance to the current third-person manipulation. For the evidence-size test, do the models' own uncertainty reports or reasoning show that they recognize the reduced precision? If not, please narrow the interpretation to stability under the displayed count transformations. Q4 Prompt reuse and model configuration — clarification. Are the medium-precision prompts in Appendix D.1 and abstract prompts in D.2 byte-identical to each other or to the corresponding core prompts? If so, were generations reused or repeated, and how does this relate to the claim that each unique prompt was queried once and to the 1,980-generation count? Please provide the hash-to-generation mapping, exact model identifiers, execution dates, system instructions, and relevant sampling/reasoning settings. This would materially improve confidence in the reported effects. Q5 Reproducible benchmark and incremental insight — clarification. Can the frozen scenario bank, actual parameter-generation and acceptance rules, all paraphrase templates, raw responses, and analysis scripts be provided anonymously during review? Appendix A describes selection criteria qualitatively and reports the resulting ranges, but does not supply all numerical rules or instantiated cases. Also, please clarify the incremental claim relative to earlier studies that vary a user's liking or disliking while holding task content fixed. I see the important extension in exact SCM-grounded interventional estimates; specifying this contribution more precisely would strengthen the paper's significance argument.