- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
- This paper investigates knowledge conflict and hallucination in LLMs using a synthetic dataset of entity–code associations.
- The authors examine how different components (QK, VO, and MLP) affect knowledge storage, retrieval, and confidence through LoRA fine-tuning.
- They analyze hidden-state geometry to explain why models can produce confident but incorrect answers.
Critical strengths
- The synthetic dataset is an interesting way to control what the model has learned and isolate different failure cases.
- The experiments are carefully conducted, covering different adapter placements, training settings, and failure modes.
- The analysis provides interesting observations on the relationship between knowledge storage, context utilization, and hallucination.
Critical weaknesses
- Several related findings have already been explored in prior work, which somewhat limits the novelty.
- [1] Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer (Dong et al., 2025).
- [2] Taming Knowledge Conflicts in Language Models (Li et al., ICML 2025).
- [3] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (Orgad et al., ICLR 2025).
- [4] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (Gekhman et al., EMNLP 2024).
- The main mechanistic experiments are conducted on a single model, making it difficult to assess how well the findings generalize.
- The reliance on synthetic entity–code associations also leaves open whether similar behaviors occur with natural-language knowledge.
Questions for authors
- Do the findings hold for other model families and sizes?
- Would similar patterns emerge when using natural-language facts rather than synthetic codes?
- How sensitive are the results to the choice of LoRA configuration?
5 comments
(Continued from Part 1/2.)
W3: Synthetic codes.
"The reliance on synthetic entity–code associations also leaves open whether similar behaviors occur with natural-language knowledge."
The conflict behavior also occurs with real-world facts that the base model already knows (rev. App. K). Note also that the scaling analysis in the paper uses natural-language benchmarks (TriviaQA, NQ-Open).
In the paper, a robust fact overrides a contradicting line in the prompt. We tested whether the base model treats its own pretrained knowledge in the same way. We took 192 facts that it answers correctly (capitals, languages, continents) and placed a line stating a wrong answer before the question. The model kept its own answer in 73% of trials and took the answer from the prompt in 25%. For our synthetic entities, about which it knows nothing, it copies the code from the prompt in 92–100% of trials. Hence the base model copies from the prompt when it has no knowledge of its own and keeps its own answer when it does, which is the behavior of our robust adapters.
We used synthetic codes as natural language does not tell us what a model has learned.
Q1: Other model families and sizes.
"Do the findings hold for other model families and sizes?"
Yes, the findings hold on a second model family (see W2). Regarding model size, the scaling result holds across 12 models from four families, and the hallucination rate across 17 models from six.
Q2: Natural-language facts.
"Would similar patterns emerge when using natural-language facts rather than synthetic codes?"
Yes, please see the response to W3.
Q3: Sensitivity to the LoRA configuration.
"How sensitive are the results to the choice of LoRA configuration?"
The same failures are observed at rank 8, 32 and 64, with no sign that they weaken at higher rank (rev. App. G, Table 16).
Fig. 7b shows that learning speed changes by at most about 15% across ranks (ln. 877–881). We have now also evaluated the behavior of the adapters from that sweep (Full placement, brittle facts).
| Rank | Recalls its facts | Copies a code from the prompt | Hallucinated answers that are another entity's trained code |
|---|---|---|---|
| 8 | 100% | 31% | 32% |
| 32 | 100% | 43% | 73% |
| 64 | 100% | 4% | 85% |
At every rank the adapter recalls its facts, copies from the prompt less often than the base model (91%), and reuses trained codes when it has no answer. Learning rate and LoRA scale are the same for all placements, so placement is the only variable in the main comparison.
With the results on a second model family, on real-world facts and across LoRA ranks, the findings are now shown to generalize, and the paper is better as a result. We thank the reviewer for suggesting these experiments! We hope we have sufficiently addressed all of the reviewer's concerns.
Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.
Thank you again for your time and consideration!
We thank the reviewer for the detailed and constructive review.
Summary of the response. The reviewer raises three concerns, namely the overlap with prior work, the single model and the synthetic facts. The revision addresses all three.
Together, these experiments substantially strengthen the thesis of the paper, as the findings now hold on a second model family, with real-world facts and across LoRA ranks. They are therefore not tied to one model, one rank or synthetic facts.
Below, "brittle" facts are those trained on a single phrasing and "robust" facts those trained on 53 (ln. 137–139). Locations given as "rev." refer to the revised manuscript.
W1: Relation to prior work.
Thank you for pointing to these works. The results are consistent with all four papers, and what this paper adds is a setting in which what the model learned is known exactly. The revision now cites [1] and [2].
W2: A single model.
Following the reviewer's suggestion, the fine-tuning experiments were repeated on a second model, Falcon3-3B-Instruct, and the same behavior is observed (rev. App. G, Table 15).
On both models, robust storage keeps recall intact and removes most or all of the copying from the prompt, brittle facts never override the prompt, and hallucinated answers reuse trained codes most often with MLP and Full adapters. One difference is that brittle adapters on Falcon leave copying intact for all placements.
In addition, the VO signature replicates across five model families (ln. 194–195, App. H), and the scaling results cover 12 models from four families.
(Continued in Part 2/2.)