acceptodds
Under review as a conference paper at ICLR 2027

True Context Corrupts Known Facts in Language Models

Abstract

We study the behaviour of LLMs on structured puzzles: each puzzle consists of a candidate answer, and a list of categories, and the LLM is asked the yes/no question of if the candidate lies in the intersection of these categories. By exploring the case wherein the candidate answer fails membership in a single ‘disqualifier’ category, but lies in the rest, we establish that LLMs are ‘distractible’: even when in isolation the models are able to correctly reject membership in the disqualifier, as we add more correct distractor categories, they progressive become more likely to incorrectly say ‘Yes,’ to the extent that with a few distractors, the majority of puzzles are answered incorrectly. This distraction is also shown to arise in free-form generation. We further establish that ‘fact corruption’ plays a non- trivial role: in certain settings, distraction arises not because the LLMs cannot conjunct category-memberships, but because they begin to endorse membership in the disqualifier. Together, this describes a new failure mode of LLMs on factual problems, and suggests a plausible mode by which hallucinations may occur.

Primary area:
interpretability and explainable AI
Keywords:
Factuality, hallucination, interpretibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bMf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes LLM hallucinations. The authors use data from a catfish game, where given a set of Wikipedia categories, the problem is to find a Wikipedia page that falls under all those categories. Using real (human submitted) wrong guesses, the paper analyzes when LLMs fail by providing a page that doesn’t fit into some of the categories (or identifying a page as belonging to all the categories).

Critical strengths
  • The problem is not particularly interesting in my opinion (find a Wikipedia page that fits all these categories), but I get that it is simple enough to dig deep and figure out what is going on. This is because at the end of the day there is (I think, I am not an expert in Wikipedia categories) no ambiguity (unlike language at large, where many factors can influence meaning).
  • I really like that the wrong guesses (negative instances) are submitted by humans, not randomly guessed or the first wrong guess by an LLM, for example.
  • The analysis provide some insights beyond when LLMs get the wrong output. For example, less informative categories (== more pages in the category) lead to more errors.
Critical weaknesses

The main problems with the paper are the rather artificial problem and that the main conclusions are already known from the literature, albeit the literature works with harder (and more interesting, in my opinion) problems rather than this very focused problem.

It is no surprise that LLMs know whether a Wikipedia page belongs to an individual category but fail to run a simple AND. There are many works that break a long input into small chunks (often referred to as “atomic”) and by manipulating the atomic facts / statements, they obtain better results than just given everything to the LLM at once. See, for example, https://aclanthology.org/2023.emnlp-main.741.pdf, https://aclanthology.org/2025.findings-naacl.158.pdf. CoT (in its simplest form) for the most part is also breaking a problem into pieces, or working through pieces of a problem to get a better answer. Based on the explanations and provided sample prompts, it looks like the authors are working with the most straightforward prompt (categories + question, zero-shot). The last experiment (5.2) basically supports this point: break the problem down into pieces and the LLM will get the right answer (with minimal prompt engineering).

It is well-established in the literature that give some factual information together with non-factual information, LLMs will just make something up that fits most of the factual stuff. See, for example, these two benchmarks and the many papers proposing approaches to stop the hallucinations: https://aclanthology.org/2023.acl-long.472/, https://aclanthology.org/2023.acl-long.583/ . I would argue that working with actual knowledge about the world and natural language is more interested than checking whether a Wikipedia page falls into some categories.

The main conclusions (based on what results they get with 3 Qwens and 1 Llama) are:

  • More true categories lead to more mistakes
  • Less informative categories (more Wikipedia pages belong to it) lead to more mistakes
  • Disqualifier placement affects error rate (it is well known that LLMs have positional biases)
  • Similar trends are observe if they follow a binary identification task or free-form generation.

These seems thin to me given the literature on this topic.

This is not a major weakness, but most formulas in Section 2 are trivial and repeat the most basic algebra of sets. It is weird to me to use space in a research paper to define unions, intersections, and set differences, especially when those symbols are not used anywhere else in the paper. It is also confusing and inconsistent. “T” is use to refer to a Wikipedia page (the Target) in 154 and as what I believe must the set of categories T belongs to in 161: “incorrect answer for a target T”). In 155, they define these categories as “C(T)”.

Questions for authors

I find some of the formulas confusing and meaningless. Isn’t the denominator in 053/054 (pi(yes)+pi(no)) just one?

Is the shading in all the plots explained somewhere? And shouldn’t we be concerned about the very high variability? I assume you run it multiple times and the LLMs are very unstable? Isn’t that a problem, given the rather large overlap in several plots?

6 comments
AuthorsOpenReview ↗

Response 1/2

Our thanks for your work. In our opinion, many of the main objections raised in the review arise from mischaracterising our paper. We hope that the following clarifies these.

The first two points are to do with the setup.

1. The setup is not artificial, and we are investigating knowledge. Wikipedia category membership is a particular way to encode factual truths. A Wikipedia page is an entity and its categories are its characteristics, stated in natural language ("Nobel laureates in Physics", "People from Ulm"). Asking whether an entity has a set of characteristics ("Is Max Planck a Nobel laureate, a 20th-century German physicist and a person from Ulm?") is one of the most common factual questions put to LLMs. This is "actual knowledge about the world and natural language". Let us also note that such category membership is factual enough for thousands of people to enjoy playing a game based on it every day!

2. The summary mischaracterises our protocols. The summary describes the task as finding a page that fits all categories. This is wrong. The principal question we investigate is whether LLMs retain a fact that it knows in the presence of other relevant and true information in the context.

In terms of the experiments, the LLM is never asked to directly solve the catfishing game. As we describe in detail in section 3.1, the main protocol is thus: for a human wrong guess W, we first verify that the model rejects W on a disqualifying category d asked alone, then ask about d together with n categories W does satisfy. The model comes to accept the answer it just rejected, more so as n grows: it asserts what it knows to be false, i.e. it hallucinates. The point of this protocol is to control ambiguity in output, and to let us easily control the main experimental variable, nn.

The next two points are to do with a mischaracterisation of our results, and of what is known from prior work.

3. Not "a failure to run a simple AND". Almost a quarter of the paper is devoted to arguing that one cannot explain away the observations in this way.

  • In Sec. 5.1, we see errors growing with nn even when we never ask for an AND, and only ask the model to resolve membership in the disqualifier.
  • In Sec. 5.2, we directly check this, by asking for an inline resolution. Here, there are two observations:
    • The error rate is nonzero, and grows with nn. We highlight this since the review wrongly says that the LLM gets the right answers in this setting! Please see Page 9, and in particular Fig. 7 and its caption.
    • Logical inconsistencies between the inline category answers and the All answer are rare. Thus, the LLMs are "running" AND correctly here, and instead, the predicate resolution itself is failing. This is interesting because we restricted ourselves to answers and disqualifiers that the model correctly resolved in isolation, because our observations are showing that true context can cause the model to fail to recognize a fact that it know in isolation.

4. Prior work. The papers cited do not imply our results at all.

  • The "atomic" papers on FActScore (Min et al. 2023) and FIRE (Xie et al. 2025) are about how to take a long LLM output, and break it into individual factual claims, which is a way these papers propose to audit LLM answers. Our experiments mostly restrict the LLM output to just Yes/No, or to few-word factual answers. These papers are about a completely unrelated problem! In fact, our work is saying that even if you know that atomic pieces in the input are right, that can still lead to increasing hallunication.
  • While CoT is indeed about breaking a problem into simpler pieces, we are not investigating CoT. In fact, for our Yes/No experiments, we are just studying a single output probability.
  • The papers on "nonfactual information" differ, indeed, by giving nonfactual information! Indeed, both Kim et al. and Yu et al. are explicitly talking about placing wrong or misleading information in the context. However, our work never does this: the disqualifier is a question that the model is verified to get correct, and all the extra categories placed in the context are true for the candidate, and our prompts are not misleading in any way.
AuthorsOpenReview ↗

Response 2/2

Finally, we discuss the review's characterisation of our main results, which too miss some of our main points.

5. The conclusions. Our main result, in a nutshell, is that adding known true facts can cause the model to endorse a false statement that, in isolation, it correctly rejected---in other words, the wider phenomenon of the distractibility, which itself has not been established (prior work instead focuses on adding misleading or wrong information). In addition, the other main result is that this effect cannot be explained away by logic errors (i.e., the model cannot compute an AND), and instead fact corruption, in the sense explored in section 5, plays at least some role. The particulars mentioned in the review are, to a degree, side results (except maybe the first, which correctly identifies the main experimental variable of number of true context examples). Naturally, we would appreciate it if the paper was judged on the basis of its actual main results.


Minor points

Notation (also Q1). Note that section 2 is giving definitions, not formulae. As with all definitions, our goal is to be precise, and ensure that the details of the work are communicated to the reader. In particular, these definitions are most extensively used in section 3.1 where we define the main experimental protocol. The notation continues to be used throughout the rest of the paper as well.

  • Specifically about π\pi on page 1: note that π\pi is said to be the LLM's output distribution. This distribution is supported on the entire vocabulary. Thus, in principle, π\pi may place mass outside of the Yes and No strings. In any case, this notation on page 1 is shorthand for the introduction. The precise definition of the score P(Yes;p)P(\texttt{Yes};p) is given in (1).

Shading and Variability (Q2). Shading was stated to be the 95% CI in the caption of Fig. 2, but we omitted this from the rest of the figures. We will correct this. In detail, in all figures, the shading represents the 95% confidence interval bootstrapped over items.

Note that the variability is over the prompts, not models. Indeed, we are directly measuring the LLM scores in the forms described, and running the same prompt twice leads to the same score, so no repetition over these is needed.

Confidence intervals for the main effects are not overlapping. Indeed,

  • In Fig. 2, the effect of interest is captured in the separation between a dashed and solid line of the same colour, and the intervals are far.
  • In Fig.s 3,4 intervals begin to overlap for large nn, but the case is already made by the separation for small nn. This is discussed in the writing.
  • In Fig.5, the left shows the soft and hard scores for the same models. Here, the main signal is the growth of these curves as nn rises.
  • Fig. 6, again, the effect of interest is the separation between dashed and solid lines of the same colour, and we have separated CIs.
  • Fig. 7 does not give CIs.
  1. I disagree, as already argued in the review. Further, calling a list of characteristics (input) and the the id/title of a Wikipedia page "natural language" is a stretch, but that's possibly a different story. Neither the input or output contain a single sentence.

  2. The paper explores that question with the task I mentioned.

  3. Breaking down the categories (we could call them atomic facts) and running an old-fashioned AND instead of letting the LLM "figure it out" would likely solve the hallucination problem in this setup, wouldn't it? You mentioned "ask for an AND" (in a prompt) and "inline resolution" (in a prompt). What I am suggesting is to not use an LLM to calculate an AND. I see no reason to use an LLM to run an AND because we can just use the actual operator (with whatever programming language). What is the conclusion, that the LLM cannot run an AND of factual characteristics? Well, running the AND is trivial in any programming language.

  4. Of course not. I never said so. All I stated is that breaking into smaller pieces (one category at a time) and combining them will solve the problem (no more hallucinated factual properties).

Reviewer_HnQkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This submission present a behavioral analysis of a particular failure mode LMs exhibit on so-called "catfishing" puzzles. These puzzles give a set of categories, such as "international rivers of Europe", "rivers of Austria", and so on, and the task is to decide if a given entity such as "Rhine" is a member of all categories or not. That is, the user or LM should answer "yes" if the conjunction of all statements "entity T is a member of the category C_i" for all i is true. The failure mode analyzed in this submission is the following: Even though an LM may give correct membership judgments for all individual categories in isolation, the model is not necessarily able to give a consistent answer when asked if the entity is a member of all listed categories. What is more, the authors find that while a model may correctly judge that an entity is not a member of all listed categories when the number of listed categories is small, adding more true categories tends to flip the initially correct judgment (i.e., "not a member") to a wrong membership judgement, possibly because the many categories in which the entity is actually a member start to outweigh, in some sense, the one category that makes the entire conjunction false. The authors try to connect this finding to issues beyond the narrow task setting and to LM hallucinations in general, but the presented arguments and evidence for this are thin.

Critical strengths

The submission identifies an interesting failure mode, and the controlled setup is well-suited for deeper analysis

Critical weaknesses

Unfortunately, the empirical and/or technical contributions are rather small.

  1. The analysis is purely behavioral and doesn't really offer any insights beyond identifying a particular failure mode in a particular setting.

  2. The authors claim that their findings bear on hallucination research and the introduction sets up an expectation that the paper will help "gain insight on hallucinations", but this expectation is never met.

  3. The paper also tries broaden its scope to "freeform generation", but the experiments in that section have a preliminary character.

To make this submission stronger, I would expect either a much more thorough behavioral analysis that leads to, if not a solution, at least to potentially actionable insights, or perhaps more interestingly, an analysis of what causes this failure mode. The interpretability literature offers multiple studies of similar failure modes, e.g., a model "knowing" a fact in one format but not in another, failures to use known facts in multi-hop reasoning, etc., some of which are already cited in the submission.

Questions for authors

What concrete insights on LLM "hallucinations" are actually gained from this paper?

3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the review.

1. Our very setting is hallucination, and we find multiple new insights on this phenomenon. "Hallucination" is an overloaded term whose meaning depends on the setting. For factual questions, a reasonable working definition is that the model gives a plausible (not random or unrelated to the given context or question) but incorrect answer.

There are three critical aspects of our experimental design that help operationalise this working definition.

  • The wrong answers we investigate are, by construction, plausible, as judged by humans: they are the most common wrong guesses submitted by players of an online quiz.
  • We verify beforehand that the model knows that the wrong answers do not fall into the disqualifier categories. This controls for missing knowledge.
  • Since everything else that is added to the prompts is true, the context is not misleading either. Further, the information is relevant to the candidate being evaluated (it lies in these classes).

Due to the first property, we are investigating hallucinations in a reasonable way, and due to the second and third, we are ensuring that the origin of these hallucinations cannot be explained away as missing context or misleading or irrelevant information placed in the context. It should be noted that both of these aspects are distinguishing from prior work (see below).

Thus, we establish the following new insights on hallucinations, namely that

  • More true context can increase hallucination: correct, relevant information makes the plausible wrong answer more likely.
  • Knowing the fact to be false does not prevent hallucination: even though we restrict to verified disqualifiers, the effect is present and strong.
  • Structural insights: the risk grows with the number of true properties the wrong answer shares with the context, and is highest when the disqualifying fact is least specific (sec. 3.3).
  • Not just logic failures, and the presence of fact corruption: the results cannot be explained away by saying that models can't perform conjunctions, since errors arise even when we only ask about the disqualifier (sec. 5.1), with no compositional question. Further, when explicitly asked to give individual memberships for all categories, logic errors (inconsistency with All answer) are absent in most settings.

Moreover, the effect seen has pertinence to free-form generation through the experiments of section 4 (more on this below).

In our opinion, these are new aspects of hallucination in factual QA that are not recognized in the literature. While we agree that actionable insights and methodologies to prevent this failure are important, these are beyond the scope of the present paper. Our focus is instead on establishing these, which itself is the first step in such directions. Indeed, how can these insights and methods be derived if the phenomenon itself is not recognized?

Prior work. Most prior work establishes and analyses hallucinations under misleading or irrelevant information. Work on contextual override uses false or irrelevant context; ours is true and relevant. Work on facts known in one format but not another compares different prompts; ours keeps the format fixed and only adds true content. Work on multi-hop failures requires isolating the facts from the query (which is hard, and often done by hand), which our design neatly avoids. Further, errors on such questions cannot be disentangled from the issue of having to compose facts, whereas we can isolate this: in sec 5.1, 5.2, nothing needs composing, yet the fact still degrades.

Further, most prior work does not validate presence of atomic knowledge, which we very explicitly do. Further, our restriction to evaluating human-plausible wrong answers is again novel.

AuthorsOpenReview ↗

Response 2/2

2. Free-form generation. We disagree with the unjustified remark that our free-form results are preliminary. sec. 4 reads, by forced decoding, the model's own probability of generating each answer. Preferring the target over a plausible wrong answer is necessary for answering correctly, so failure here implies failure in free generation (Fig. 5).

Perhaps it is the case that this feels preliminary because we do not sample the answers, and instead force to the two answers? To ameliorate this, we have conducted a sampling based study, described below.

We sampled 50 free-form answers per prompt at temperature 1, over 13,334 prompts for Qwen-7B and 31,116 for Llama-8B (about 0.67M and 1.56M answers, all three disqualifier positions; half are reference prompts), and report the wrong answer's share among answers naming either the target or the wrong answer. As an additional control, we also ran each prompt with the disqualifier replaced by one more shared category, so that the wrong answer is also valid.

There are two salient observations:

  1. as nn rises, the raw empirical probability of outputting the wrong answer rises from 3.5% (n=1n = 1) to 16% (n=7n = 7) for Qwen-7B, and from 1.9% (n=1n= 1) to 11% (n=7n = 7) for Llama-8B. Thus, in absolute terms, the rate of generating the wrong answer rises about 5 times.

  2. The rate relative to the target (i.e., π^(W)/(π^(T)+π^(W))\hat{\pi}(W)/(\hat{\pi}(T) + \hat{\pi}(W)) ) rises by about 2 times in the presence of a disqualifier, but stays flat when we replace by the control with shared category.

     Qwen2.5-7B                  Llama-3.1-8B
 n     disq.     shared  pairs     disq.     shared  pairs
 1     0.14      0.58    2,534     0.09      0.58    4,497
 4     0.27      0.56      461     0.19      0.53    1,636
 7     0.35      0.56       72     0.23      0.51      552

3. Analysis of what causes the failure. Let us note, again, that the paper does analyse what causes the failure in a behavioural sense: it is the presence of further true information introduced into the context.

Perhaps the intention here was to focus on mechanistic explanations? Towards this end, we performed the following experiment to argue that the true categories outweigh the disqualifier in a causal way.

  • Setup: Let w(i,j)w(i,j) be the attention weight from position ii to token jj. Let LL be the list of category name tokens, and let D⊂LD \subset L be the disqualifier name tokens in this list. Note that since the names have multiple tokens, ∣D∣>1|D| > 1 even though we only look at single-disqualifier prompts.

  • Experiment: For every position ii after the list, we intervene on the attention weights for these tokens in every head and layer. For a parameter α,\alpha, this intervention is thus:

    w(i,j) & \textrm{otherwise} \end{cases}, \\\\ \textrm{where } f_i := \frac{\sum_{L} w(i,j)}{\sum_{L \setminus D} w(i,j) + \sum_D \alpha w(i,j)}.$$ - In words: - We keep the total attention placed on all the category name tokens fixed (hence the choice of $f_i$) - We either upweigh the attention placed on the disqualifier $(\alpha > 1)$ or downweigh it $(\alpha < 1)$ relative to the rest of the category-name tokens. - Notice that if we can attribute the effect to overweighing of the distractors then we expect two things: - For prompts that the LLM got wrong, if we set $\alpha > 1$, we should see a decrease in errors. - For prompts that the LLM got right, if we set $\alpha < 1$, we should see an increase in errors.
  • Observations: the main observations bear these predictions out

    • More weight on the disqualifier repairs errors. We choose 300 wrong prompts randomly from the set which has very high P(Yes). On these, setting α=8\alpha = 8 leads this P(Yes) to fall from 99% to 37% (Qwen-7B) and 61% (Llama-8B). If we instead boost a distractor by the same amount, we see error rates of 8484 and 9292 percent. It is interesting that this is still a decay, but nevertheless note that this is much higher than when the disqualifier is boosted.

    • Less weight creates errors: Similarly, we picked 300 correct prompts out of a set with very small P(Yes). Here, P(Yes) rises from 3% to 38% (Qwen-7B) and from 1% to 52% (Llama-8B) at α=0\alpha = 0; removing a distractor category instead leaves 3% and 5%.

These results, while preliminary, indicate that the knowledge is still present and usable, but is outweighed in context by true supporting content. We will further develop and add this analysis to the paper, if accepted.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer HnQk,

Thank you again for your review. As the discussion period ends on Nov 18, we wanted to check whether our response addresses your concerns. In particular, following your suggestion, we added an analysis of what causes the failure (an attention intervention) and a direct free-form generation experiment. We would be grateful for your feedback and are happy to clarify anything further.

Reviewer_J5SaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a substantial, interpretable behavioral effect and supports it with useful selection and confidence checks. However, the evidence currently establishes a narrow conjunction inconsistency more convincingly than factual corruption or increased free-form hallucination. Matched controls, independent label auditing, direct generation evaluation, and reproducible dependence-aware analysis would materially strengthen the case for acceptance.

Critical strengths
  • Interpretable test of compositional inconsistency. The design explicitly verifies that a candidate is rejected under its isolated disqualifier before evaluating the conjunction. Thus, the failures cannot simply be attributed to never having elicited the relevant negative judgment. Reducing composition to conjunction makes the resulting inconsistency particularly interpretable and provides a useful experimental setting for studying factual reliability.

  • Large effect across model sizes and families. Mean Yes probability and thresholded false acceptance both increase substantially, with mean probability exceeding 0.5 by six distractors across the four tested models

Critical weaknesses

Negative padding is not a matched test of truth-specific distraction. The main condition adds satisfied predicates while retaining one failed predicate; the control adds further failed predicates. These interventions differ in evidence supporting rejection. Declining acceptance with additional disqualifiers therefore does not isolate length from affirmative-evidence accumulation.

Questions for authors
  1. Does the central trend persist under semantically matched neutral padding and redundant positive predicates, with the same targets, wrong answers, and disqualifiers?
  2. What proportion of sampled disqualifiers are independently false semantic properties, rather than merely absent direct Wikipedia category assignments? Please report a manual audit and membership/snapshot conventions.
3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the careful review. Below, we first address the two main questions that the review asks. Following that, we also include further argument regarding some points made in the summary that we disagree with.

  1. Neutral padding and redundant predicates. We carry both of these out below.

Before presenting results, let us note that it is not a priori clear what a neutral padding is: any category we add will either include the candidate, or not. That said, random choices will very likely be negative, so your point is well taken. As a way of getting to a purely 'neutral' padding, we use length controlled whitespaces. Specifically, we replace the distractor categories with whitespaces (i.e., ' ') matching the replaced distractors in token length. This lets us control for the potential effect of just a longer or shorter pad, while not attaching a true/false status. In the repeated category experiment, we pick one distractor category, and repeat it nn times. Below, we report the mean P(Yes) numbers under these settings, along with the original.

     Qwen2.5-7B                   Llama-3.1-8B
 n   orig.  repeat  neutral       orig.  repeat  neutral
 1   0.24   0.23    0.04          0.49   0.49    0.15
 3   0.52   0.51    0.02          0.75   0.63    0.18
 5   0.55   0.50    0.01          0.82   0.62    0.19
 7   0.58   0.53    0.00          0.85   0.60    0.16

(each cell has ≥238\ge 238 pairs.) There are two salient observations: 1) repeated true content corrupts the fact fully for Qwen and partly for Llama. Thus, even a single category, stated many times, can lead to distraction, although more work is needed to understand if this mode leads to saturation more quickly, and if the net effect size is diminished (which we will add, if accepted), and 2) neutral padding by whitespace again does not show a corresponding increase, bearing out that a true positive distractor categories are needed.

  1. Independent audit of disqualification. Let us first note that our measure does not depend on Wikipedia's tagging: we keep only items where the model itself rejects the disqualifier alone (sec. 3.1), so rises in P(Yes) show that the model contradicts its own judgement either way.

Nevertheless, the point is well taken. As suggested, we carried out a manual audit of a 100 wrong answer and disqualifier category pairs (restricting to categories that the LLM rejected). Herein, we used manual wikipedia and google searches to try to disprove membership, each time confirming the fact by confirming in the cited sources. Here we found that in 93 instances, the disqualification is correct, in 5 it is incorrect (but the tag is missing in the wikipedia categories) and 2 were ambiguous. This suggests that, the vast majority of the disqualifications are correct (99% Clopper-Pearson CI is (0.83, 0.98) ).

In case the questions about membership conventions and snapshot were about the paper, this was discussed, but nevertheless: the membership convention is simply presence in the category pages (so, weaker than the manual audit above), and the instances are built out of catfishing puzzles numbers 32 through 806. All of these were scraped in September 2026.

AuthorsOpenReview ↗

Response 2/2

  1. Dependence-aware analysis. The review suggests a reproducible, dependence-aware analysis. Our analysis already is, throughout the paper: duplicate target-candidate pairs are dropped (earliest puzzle kept); each pair contributes one value (positions averaged within a set, sets within a pair); 95% CIs are cluster bootstrap over puzzles (1,000 resamples); all sampling uses fixed seeds. We will detail this in the appendix.

  2. Beyond a conjunction inconsistency. We would like to highlight that the paper goes beyond only establishing the inconsistency through the experimentation in section 5, where we establish that even when we ask the model to resolve only membership in the disqualifier, we see an increase in errors as the number of padded categories increases. In fact, in the version of the experiments in Sec.5.2, we see next to no conjunction inconsistency (in terms of the explicit inline responses and the all response) in most cases, but instead see errors arising from the disqualifier being resolved incorrectly. In the version of sec.5.1, there is no reference to a conjunction at all, but still the wrong answer rate rises from near zero to about 0.15.

In our response to reviewer HnQK, we also give further mechanistic analysis: keeping the question's total attention to the list fixed and raising only the disqualifier's share cuts errors on failed prompts from 99% to 37% (Qwen2.5-7B) and 61% (Llama-3.1-8B); lowering it on correct prompts raises errors from 3% to 38% and from 1% to 52%.

  1. Free-form generation. The review finds the evidence for free-form hallucination less convincing and suggests a direct generation evaluation. In our opinion, sec. 4 already measures generation: we read, by forced decoding, the model's own probability of generating the target and the wrong answer. Preferring the target over a plausible wrong answer is necessary for answering correctly, so failure on this measure implies failure in free generation.

To show that the failure is also directly present in free-form generation, we carried out the following further experiment: we sampled 50 free-form answers per prompt at temperature 1, over 13,334 prompts for Qwen-7B and 31,116 for Llama-8B (about 0.67M and 1.56M answers, all three disqualifier positions; half are reference prompts), and report the wrong answer's share among answers naming either the target or the wrong answer. As an additional control, we also ran each prompt with the disqualifier replaced by one more shared category, so that the wrong answer is also valid.

There are two salient observations:

  1. as nn rises, the raw empirical probability of outputting the wrong answer rises from 3.5% (n=1n = 1) to 16% (n=7n = 7) for Qwen-7B, and from 1.9% (n=1n= 1) to 11% (n=7n = 7) for Llama-8B. Thus, in absolute terms, the rate of generating the wrong answer rises about 5 times.

  2. The rate relative to the target (i.e., π^(W)/(π^(T)+π^(W))\hat{\pi}(W)/(\hat{\pi}(T) + \hat{\pi}(W)) ) rises by about 2 times in the presence of a disqualifier, but stays flat when we replace by the control with shared category.

     Qwen2.5-7B                  Llama-3.1-8B
 n     disq.     shared  pairs     disq.     shared  pairs
 1     0.14      0.58    2,534     0.09      0.58    4,497
 4     0.27      0.56      461     0.19      0.53    1,636
 7     0.35      0.56       72     0.23      0.51      552
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer J5Sa,

Thank you again for your review. Our response addresses the questions you raised (matched controls, a manual label audit, direct generation, and dependence-aware analysis), and we would be grateful for any feedback before the discussion period ends on Nov 18. We are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their time and feedback. We first restate what the paper studies, since it is central to several comments, and then summarize the further analyses we ran in response to the reviews.

What the paper studies

We investigate hallucination by studying the response of LLMs to plausible but incorrect answers. The wrong answers we use are, by construction, human-plausible: they are the most common wrong guesses submitted by players of an online quiz. For each wrong answer W, we first verify that the model rejects it on a disqualifying category d when asked about d alone. We then ask about d together with n categories that W does satisfy. As n grows, the model not only increasingly accepts the conjunction of these category memberships, but also increasingly accepts that the answer lies in the disqualifier, even though it rejects this in isolation.

This isolates a particular mode of hallucination which, to our knowledge, has not been studied. Here, hallucination is not arising due to missing knowledge, since the fact is verified beforehand. It is also not arising due to false or irrelevant context, since every distractor category in the prompt is true and relevant. And it is also not purely a failure to combine category-verdicts (i.e., failure to AND): when asked about d alone with the other categories in context, the model comes to endorse d (sec. 5.1), and when it gives a verdict on each category first, almost every error is a wrong verdict on d itself (sec. 5.2). Thus, we establish that true context alone can corrupt a known fact.

Further analyses

In addition to the rebuttals on the basis of the text, we have run the following further analyses in response to specific concerns raised in the reviews. For the sake of time, these are all run only on Qwen-2.5-7B and Llama-3.1-8B, but we are committed to running them for other models in the future.

  1. What causes the failure. HnQk noted that the analysis is behavioral and asked what causes the failure. We performed an attention intervention that keeps the total attention to the category list fixed and changes only the disqualifier's share. It reverses the behavior in both directions. More weight on the disqualifier cuts errors on failed items from 99% to 37% (Qwen-7B) and 61% (Llama-8B); less weight raises errors on correct items from 3% to 38% and from 1% to 52%. The same boost on a supporting category has little effect. The fact is known, but outweighed by true supporting content.

  2. Content, not length. J5Sa asked whether the trend persists under length-matched neutral padding and redundant positive predicates; bMf3 suggested the effect may come from mixing in non-factual information. Length-matched neutral padding does not raise P(Yes) with n, while true categories raise it from 0.24 to 0.58 (Qwen-7B) and from 0.49 to 0.85 (Llama-8B). Even a single true category repeated n times raises it well above the neutral control.

  3. Direct generation. J5Sa found the evidence for increased free-form hallucination less convincing and suggested a direct generation evaluation; HnQk described the free-form experiments as preliminary. We believe our results are neither preliminary nor less convincing. The free-form experiment (sec. 4) reads the model's own probability of generating the target and the wrong answer. Preferring the target over a plausible wrong answer is necessary for answering correctly, so failure on this measure is sufficient to conclude that free-form generation fails as well. Nevertheless, we understand that direct generation experiments may be preferable.

To show that the failure is also directly present in free-form generation, we sampled 50 free-form answers per prompt at temperature 1, over 13,334 prompts for Qwen-7B and 31,116 for Llama-8B (about 0.67M and 1.56M answers, all three disqualifier positions). Herein, the absolute rate of giving the wrong answer rises roughly 4.5×4.5\times as nn goes from 11 to 77: from 3.5% to 16% for Qwen-7B, and 1.9% to 11% for Llama-8B. Further, when restricted to T or W, the error rate more than doubles over the same range: 0.14→0.350.14 \to 0.35 for Qwen, and 0.09→0.230.09 \to 0.23 for Llama. Finally, this increase does not occur if the disqualifier is replaced by a shared category. The sampled shares and the probabilities of sec. 4 agree prompt by prompt (Spearman ρ ≈ 0.75).

  1. Disqualifier audit. J5Sa asked how many disqualifiers are false properties rather than missing category tags. In a manual audit of 100 disqualifiers, 93% are false properties of the wrong answer, 5% hold but lack a direct category assignment, and 2% are ambiguous.

  2. Dependence-aware analysis. J5Sa asked for an analysis that accounts for dependence between items. We describe the de-duplication, per-pair aggregation and cluster bootstrap over items used throughout the paper.

We will add all of these analyses to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.