- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces UNSTATED, a benchmark for identifying a failure mode of large language models that the authors call norm-to-fact projection: when a text states only a rule and says nothing about the outcome, does the model treat the rule as evidence of whether the event actually happened?
The benchmark was constructed by five human experts and contains 120 scenarios, each instantiated under six conditions, for 720 items in total. Conditions A (prohibition) and D (command) are the unstated-outcome probes: they give only the rule, with no factual outcome, so the correct label is neutral. Conditions B, C, E, and F are the explicit-outcome anchors: they give the same rule together with an explicit statement of whether the event occurred.
The authors evaluate 28 open-weight models and 10 frontier models. The main finding is that open-weight models perform reasonably well on the explicit-outcome anchors (median accuracy 84.5%) but make strongly directional errors on the unstated-outcome probes: prohibitions are read as "did not occur" and commands as "occurred" (median accuracy 6.7% on the prohibition probe and 15.8% on the command probe). Frontier models perform better overall, but their median accuracy on the prohibition probe is still only 74.2%, with large variation across models (20.0%–98.3%).
Critical strengths
- Clean, well-controlled design. Within each scenario, only the outcome information changes across conditions, while participants, events, and hypotheses are held fixed. The matched explicit-outcome anchors separate the target error from general difficulties with factual recognition and NLI label mapping, and the additional controls (affirmative requirements versus negated prohibitions, unrelated-hypothesis fillers, three contextual frames) rule out several simple alternative explanations.
- A clear and practically relevant error pattern. The errors are strongly directional: prohibitions lead to "did not occur" and commands lead to "occurred". Because the two probes fail in opposite directions, the errors are tied to the type of rule rather than to a generic label preference. The errors are also made with high confidence (probe predictions with ≥90% confidence are only 22.5% accurate), which suggests that, at least for open-weight models, confidence-based triage would not catch them.
My main concern was that this benchmark measures only an input-sensitive failure mode that arises at lower levels of model capability. I believe the authors have addressed this concern through their responses in A1-1 and A1-5. Therefore, I have decided to raise my score.
Critical weaknesses
1. The headline error rates are not shown to reflect model capability rather than the evaluation protocol and task format.
Open-weight models are evaluated with restricted-choice readout over the next-token probabilities of A/B/C, with native thinking modes explicitly disabled, whereas frontier models generate a response that is then parsed; the paper only states that frontier models used "default decoding settings" and does not report whether reasoning was enabled or at what effort level. The open-weight versus frontier comparison therefore conflates readout protocol, reasoning, and model scale. More importantly, the error is highly sensitive to format and prompting:
- Under a generic "Let's think step by step" CoT prompt (seven models, exploratory), median command-probe accuracy rises from 14.2% to 71.7% and prohibition-probe accuracy from 10.0% to 34.2%. For Qwen3.5-9B, from a model family that offers a native thinking mode (disabled in the main evaluation), prohibition-probe accuracy rises from 7.5% to 68.3%.
- For frontier models, a single added reminder (P2: a rule describes what should happen) raises median prohibition-probe accuracy by 25.8 percentage points, while anchor accuracy drops by only 2.3 points.
- In Appendix M.1, simply rephrasing the question as "which outcomes remain possible" raises the prohibition-probe accuracy of Qwen3.5-9B from 7.5% to 50.8% (although this model narrowly misses the authors' own task-control threshold in that experiment).
There is also no downstream task showing that the error causes actual losses in realistic use; the three regulatory cases in Appendix L are illustrative and still in NLI format.
Why this is consequential: The paper's central claim is that norm-to-fact projection is "a systematic and underexamined failure mode in current LLMs". The evidence above suggests that much of the measured effect may be induced by a specific setup (single-sentence three-way NLI, restricted-choice readout, disabled reasoning) and can be substantially reduced by a one-sentence instruction, at least for frontier models. Without evaluation under native reasoning and evidence from a realistic task, the significance of the finding is unclear, and this is the main reason for my recommendation.
2. The phenomenon is not shown to be specific to normative rules, so the main contribution is not well established.
The paper names the error norm-to-fact projection and attributes it to normative operators, yet the authors' own results show that a normative operator is not necessary. In the operator ablation (Table 27), when no rule is given and the premise only says that the person "had never shown any inclination to" perform the act, the median rate of "did not occur" outputs is 100.0%, at least as high as the 96.7% obtained with the prohibition alone; adding "the temptation was overwhelming with no one nearby to intervene" to the prohibition lowers it to 55.4%. In Appendix J.7, some non-deontic expressions also trigger directional completion: ability statements are read as "occurred" at a median rate of 84.2%, and low-likelihood statements ("Likely low" in Table 30) are read as "did not occur" at 98.8%, slightly higher than the 95.8% for prohibitions. The authors note that "default-outcome completion extends beyond deontic vocabulary but does not produce a uniform response to every modal expression." Although the authors describe the rule-free rate in Table 27 as "only an output rate and not a norm-to-fact projection measure", both conditions have neutral gold labels and measure the same output. Together, these results suggest that the outcome's prior plausibility drives the outputs to a large extent, with the rule being one source of that prior.
These appendix results bear directly on the scope of the phenomenon, but they are not discussed in the main text, which continues to frame the error as specific to rules. In addition, the mechanism studies in Appendix M are behavioral, and the authors acknowledge that "training-stage and representational mechanisms remain unresolved"; the prompt interventions are reported to "not establish a reliable general repair."
Why this is consequential: The first stated contribution is to identify norm-to-fact projection as a failure mode. If the rule has no effect beyond shifting the outcome prior, the paper may be documenting a specific instance of a more general, prior-driven behavior rather than a distinct failure mode, and readers gain little insight into its cause or remedy.
Questions for authors
- Native reasoning. Can the open-weight models be re-evaluated with their native thinking modes enabled (e.g., the
<think>mode of the Qwen3 and Qwen3.5 series), reporting probe and anchor accuracy and comparing them with the restricted-choice readout? If the gap between the two groups shrinks substantially once reasoning is enabled, I would conclude that the main findings reflect the evaluation protocol more than model capability. - Frontier-model settings. Can the authors report, for each frontier model, temperature, top_p, max_tokens, and whether reasoning/thinking was enabled and at what effort level?
- Downstream relevance. Can the authors show, on at least one real downstream task (e.g., compliance checking, contract review, or process auditing), that (a) norm-to-fact projection is positively correlated with the downstream error rate, and (b) simple prompt engineering (e.g., adding "judge only from evidence explicitly stated in the text") cannot reliably fix it?
- Role of the rule. Given that in Table 27 the rate of "did not occur" outputs reaches 100% when no rule is given and only a low-probability cue is present, to what extent does the rule have an effect independent of the outcome prior? Can the authors compare matched conditions with and without a rule while holding the prior plausibility of the outcome fixed? The answer determines whether norm-to-fact projection stands as a distinct failure mode.
8 comments
Response to Reviewer th9p (Part 2)
Q1-3: There is also no downstream task showing that the error causes actual losses in realistic use;
A1-3: This is an interesting question. UNSTATED scenarios draw on requirements, prohibitions, and actions in everyday life, rather than symbolic logic without practical context. Our experiments show norm-to-fact projection in both locally evaluated open-weight models and API-evaluated frontier models. Prompting and CoT can improve performance, but this risk remains, indicating that current LLMs may make incorrect judgments when interpreting practical texts. To further address your concern and examine downstream effects, we collected 100 audit units from the official 2024 Code of Federal Regulations (CFR): 50 without execution evidence and 50 with explicit evidence. We evaluated Llama-3.1-8B, Qwen3.5-9B, GPT-5.5, and Claude Opus 5, all with thinking disabled. The results are shown below:
| Model | No execution evidence: “cannot determine” (correct) | No execution evidence: rule-consistent error | No execution evidence: judgment against the rule | Explicit evidence: accuracy |
|---|---|---|---|---|
| Llama-3.1-8B | 6.0 | 82.0 | 12.0 | 74.0 |
| Qwen3.5-9B | 8.0 | 88.0 | 4.0 | 92.0 |
| GPT-5.5 | 92.0 | 6.0 | 2.0 | 96.0 |
| Claude Opus 5 | 56.0 | 40.0 | 4.0 | 90.0 |
These results show that treating rules as facts also occurs in a real downstream task, highlighting the practical value of studying this error. We will include this experiment in the next version.
Q1-4: the three regulatory cases in Appendix L are illustrative and still in NLI format.
A1-4: First, Appendix L uses the NLI format to align with the original benchmark, allowing us to test under the same evaluation criteria whether confusion between rules and facts also occurs in real regulatory text. Second, NLI is a well-established task in computational linguistics, with widely used evaluation resources such as SNLI (EMNLP 2015) [5] and MultiNLI (NAACL 2018) [6]. Its entailment, contradiction, and neutral relations clearly distinguish facts supported by the text from outcomes the text does not determine. This directly matches our goal of testing whether models treat normative requirements as factual evidence.
[5] Bowman et al. (2015). A large annotated corpus for learning natural language inference. EMNLP.
[6] Williams et al. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. NAACL-HLT.
Q1-5: Why this is consequential: The paper's central claim is that norm-to-fact projection is "a systematic and underexamined failure mode in current LLMs". The evidence above suggests that much of the measured effect may be induced by a specific setup (single-sentence three-way NLI, restricted-choice readout, disabled reasoning) and can be substantially reduced by a one-sentence instruction, at least for frontier models. Without evaluation under native reasoning and evidence from a realistic task, the significance of the finding is unclear, and this is the main reason for my recommendation.
A1-5: As discussed in Q1-1–Q1-4, we first explain the basis for our evaluation protocol and NLI design. The CoT and direct outcome-question experiments show that prompting and reasoning can improve performance, but some models still show clear confusion between rules and facts. In Q1-3, we further evaluate 100 audit units from real documents and show that the error also occurs in this setting. To further address this concern, we evaluated Qwen3 and Qwen3.5 with native <think> mode enabled and compared them with CoT prompting. Each cell reports CoT prompt (P1gen) / native Think accuracy (%):
| Model | Overall | Acc_A | Acc_D | Anchor |
|---|---|---|---|---|
| Qwen3–14B | 84.9 / 92.1 | 38.3 / 77.5 | 85.8 / 99.2 | 96.3 / 94.0 |
| Qwen3–32B | 86.9 / 90.6 | 41.7 / 70.0 | 95.8 / 99.2 | 96.0 / 93.5 |
| Qwen3.5–4B | 84.9 / 87.5 | 45.0 / 55.8 | 85.8 / 96.7 | 94.6 / 93.1 |
Native Think further improves probe performance for these models, but prohibition-probe accuracy remains only 55.8–77.5%, well below explicit-outcome anchor accuracy of 93.1–94.0%. Thus, these models still do not reliably distinguish rules from facts with native reasoning enabled. Thank you for this comment. We will include this experiment in the next version.
Response to Reviewer th9p (Part 3)
Q2-1: The phenomenon is not shown to be specific to normative rules, so the main contribution is not well established. The paper names the error norm-to-fact projection and attributes it to normative operators, yet the authors' own results show that a normative operator is not necessary.
A2-1: Our paper starts with a specific question: do models mistake “what should happen” for “what actually happened”? By pairing unstated-outcome probes with explicit-outcome anchors, UNSTATED systematically shows that models can identify explicit facts yet still treat normative requirements as outcome evidence (Table 2). This behavior has a reasonable link to outcome expectations: Gächter et al. (2025) [7] find that people expect others to follow rules, while Phillips and Cushman (2017) [8] show that norms affect default judgments of what is possible. Building on this link, we compare rules, behavioral propensity, and likelihood cues in Appendices J.3 and J.7, finding that non-rule cues can also lead to similar errors. These extended results do not contradict the original finding; they connect confusion between rules and facts to a broader expectation-driven problem. Our contribution is both to identify and measure this specific error and to test its scope through further controls. It does not require rules to be a necessary condition for all similar errors.
Q2-2: In the operator ablation (Table 27), [... ...]. Together, these results suggest that the outcome's prior plausibility drives the outputs to a large extent, with the rule being one source of that prior.
A2-2: Prior work provides empirical evidence for a link between rules and outcome expectations. Gächter et al. (2025, Nature Human Behaviour) find that rules lead people not only to judge that others should comply [7], but also to expect that others will comply. Phillips and Cushman (2017, PNAS) further show that norms affect default judgments of what is possible [8]. Szekely et al. (2021) find that both normative expectations and expectations about others' behavior affect cooperative decisions [9]. Bonan et al. (2020) show that agreement or conflict between normative requirements and information about actual behavior changes intervention effects [10]. These studies motivate our comparisons of rules, behavioral propensity, and likelihood cues. A prior-driven account is therefore compatible with norm-to-fact projection: the former provides an account of how the latter arises.
[7] Gächter et al. (2025), Why people follow rules, Nature Human Behaviour
[8] Phillips, J., & Cushman, F. (2017). Morality constrains the default representation of what is possible. PNAS, 114(18), 4649–4654.
[9] Szekely et al. (2021). Evidence from a long-term experiment that collective risks change social norms and promote cooperation. Nature Communications, 12, 5452.
[10] Bonan et al. (2020). The interaction of descriptive and injunctive social norms in promoting energy conservation. Nature Energy, 5, 900–909.
Q2-3: These appendix results bear directly on the scope of the phenomenon, but they are not discussed in the main text, which continues to frame the error as specific to rules.
A2-3: The main text discusses the link between rules and outcome expectations in Section 3.3 (lines 215–219). We have also added §5.5, “Do Outcome Expectations Shape Factual Judgments?”, to summarize the results from Appendices J.1, J.3, and J.7 and discuss what they imply for the scope of the phenomenon. All added and revised text is marked in blue.
Q2-4: In addition, the mechanism studies in Appendix M are behavioral, and the authors acknowledge that "training-stage and representational mechanisms remain unresolved"; the prompt interventions are reported to "not establish a reliable general repair."
A2-4: Yes, we have not established the training-stage or representational mechanisms behind this behavior, nor proposed a reliable general repair. Our paper defines and measures the error through paired experiments, tests it across models and contextual frames, and examines factors that affect it through controls and prompt interventions. This fulfills the goal of UNSTATED as a diagnostic benchmark: to provide a reproducible basis for identifying, comparing, and studying this problem. A more complete explanation of the mechanisms and targeted ways to reduce the error remain important directions for future work; our analyses provide evidence for these efforts.
Response to Reviewer th9p (Part 1)
Q1-1: The headline error rates are not shown to reflect model capability rather than the evaluation protocol and task format.Open-weight models are evaluated with restricted-choice readout over the next-token probabilities of A/B/C, with native thinking modes explicitly disabled, whereas frontier models generate a response that is then parsed; the paper only states that frontier models used "default decoding settings" and does not report whether reasoning was enabled or at what effort level. The open-weight versus frontier comparison therefore conflates readout protocol, reasoning, and model scale.
A1-1: This reflects a misunderstanding of our evaluation design. First, all models are evaluated on the same three-way NLI task. For open-weight models, we select the highest-probability option, following prior work [1–2], and retain option probabilities for confidence analysis. For API-evaluated models, we parse their final answers into labels because complete, comparable option probabilities are not available in our evaluation. Second, to further address this concern, we switched open-weight models to generated answers while keeping the other output settings unchanged. The table reports medians across 28 models:
After aligning the open-weight readout with that of frontier models, overall performance is slightly lower rather than higher, and the overall pattern remains consistent. These results support our choice of evaluation protocol and indicate that the gap from frontier models reflects model capability rather than the readout protocol.
[1] Lost in Inference, NAACL 2025 Main
[2] A Rose by Any Other Name, Findings of ACL 2025
Q1-2: More importantly, the error is highly sensitive to format and prompting:Under a generic "Let's think step by step" CoT prompt (seven models, exploratory), median command-probe accuracy rises from 14.2% to 71.7% and prohibition-probe accuracy from 10.0% to 34.2%. For Qwen3.5-9B, from a model family that offers a native thinking mode (disabled in the main evaluation), prohibition-probe accuracy rises from 7.5% to 68.3%. For frontier models, a single added reminder (P2: a rule describes what should happen) raises median prohibition-probe accuracy by 25.8 percentage points, while anchor accuracy drops by only 2.3 points. In Appendix M.1, simply rephrasing the question as "which outcomes remain possible" raises the prohibition-probe accuracy of Qwen3.5-9B from 7.5% to 50.8% (although this model narrowly misses the authors' own task-control threshold in that experiment).
A1-2: First, we tested a range of alternative explanations under direct answering, supporting the validity of the low performance measured in this setting. Second, large gains from CoT have also been reported in prior work. Both the original CoT paper [3] and the “Let's think step by step” paper [4] report substantial gains across several datasets; for example, CoT raises MultiArith accuracy from 7.3% to 73.7%. A large CoT gain alone therefore does not establish that our evaluation is unusually sensitive to prompting. Finally, although prompting and reasoning can improve performance, our benchmark remains challenging. With CoT, the seven models still have median error rates of 65.8% on prohibition probes and 28.3% on command probes, leaving accuracy gaps of 64.3 and 25.3 pp from the human baseline, respectively (Table 3 and Appendix K). Even Qwen3.5-9B, which improves substantially, still makes errors on 31.7% of prohibition probes and remains 30.2 pp below human accuracy. Large gains therefore do not mean that the problem is solved: reliably distinguishing normative requirements from factual outcomes remains difficult.
[3] Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS.
[4] Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS.