acceptodds
Under review as a conference paper at ICLR 2027

Teaching LLMs New Knowledge Without Hallucinations

Abstract

Large language models increasingly operate in settings where they encounter information that was unavailable during pretraining. While such knowledge can be stored externally, continually internalizing it into model weights could improve efficiency and enable reasoning across information acquired at different times. We study knowledge acquisition from natural-language documents and identify a central failure mode: fine-tuning on new knowledge can rapidly increase hallucinations. Models become more willing to answer questions about the newly learned domain even when the required information was never provided, and newly acquired facts can also spill over into answers to unrelated questions. Such hallucinations are undesirable in themselves and are especially concerning in continual-learning systems, where they may accumulate over time. We introduce hallucination-aware training, which combines context distillation with hallucination-probing questions that teach the boundary of the acquired knowledge and a regularization objective that preserves the model’s behavior on unrelated questions. Across Qwen3-32B, GPT-OSS-20B, Gemma4-31B, and Qwen3.8-27B, our approach substantially reduces hallucinations and knowledge spillover while preserving strong acquisition of new knowledge. We further find that models can compose facts learned independently from different documents, even though training processes only one document at a time and provides no explicit supervision for cross-document composition. Our results suggest that LLMs can internalize new knowledge while learning both how to use that knowledge and where its boundaries lie.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, fine tuning, hallucination mitigation, knowledge acquisition, knowledge internalization, knowledge aggregation, context distillation, continual learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LweUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a benchmark, a fine-tuning method, and an analysis of the problems smaller LLMs run into when they learn new knowledge. The study uses four models and two fictional company databases, each made up of several documents (four databases in the scaling experiments). The benchmark measures four things. The first is accuracy on the new facts. The second is how free of hallucination the learning was, tested with questions whose answers cannot be known. The third is how accuracy on old knowledge compares. The fourth is whether answers to old-knowledge questions stay unrelated to the fictional companies, where a high rate means little knowledge spills over. The authors' hallucination-aware training improves on all of these metrics. It combines context distillation with hallucination-probing questions and old-knowledge regularisation, which makes the models more reliable at the boundaries of what they know.

Critical strengths

The paper is well written. The figures, tables and sections all support the argument and each one is needed. The method does more than reduce hallucinations. It focuses on the boundary of knowledge ingestion in LLMs. Rather than only asking whether new knowledge can be retrieved, the evaluation covers several angles: old-knowledge accuracy, transfer across documents, behaviour on old knowledge, and compositional accuracy. The paper also gives its prompts, including those for the LLM-judge evaluations, in a thorough appendix. I also appreciate that the authors go past their registered metrics and compare the grounding of the teacher.

Critical weaknesses

My main concern is the number of questions and repetitions. The main experiment has 20 questions for each of the three types, 10 per company across 2 companies, so each question is worth 0.05 accuracy. The paper reports no training seeds, no repeated runs and no measure of spread. With accuracy this coarse, the differences between conditions are hard to interpret. For example, the gap between KL and SFT is small compared with the weight of a single question. A related point: it is not clear to me how much the training probes and the test probes overlap. The model might simply have learned to decline questions of a familiar shape.

The related work section lists many methods and compares them with the proposed approach in conceptual terms, but none of them is evaluated against it on the paper's metrics. The main baseline the authors introduce is a variant of their own distillation.

Even with the extensive appendix, several prompts are missing: the prompts for fact extraction, question generation, and unanswerable and false-premise generation, the teacher's instructions, and the recall-first instruction that separates Grounded+ from Grounded−.

Please also check these smaller issues:

  • Section 5.4 says "all three models" twice (L383, L393–394), but Fig. 2 shows four.
  • The Gemma4-31B row of Table 3 adds up to 35,738, but the table gives 45,738 (L680).
Questions for authors
  1. How many training seeds or repeated runs are behind the reported numbers, and how much do the results vary between them? With 20 questions per type, each question is worth 0.05 accuracy, so it would help to know whether differences such as the KL vs SFT gap are larger than this spread.
  2. How much do the training probes and the test probes overlap? Could the model have learned to decline questions of a familiar shape instead of recognising the limits of its knowledge? Results on test probes whose form differs from the training probes would help answer this.
  3. Why are the methods discussed in the related work not evaluated as baselines on your metrics? Apart from the variant of your own distillation, is there an external method you could compare against?
  4. Could you add the missing prompts to the appendix: fact extraction, question generation, unanswerable and false-premise generation, the teacher's instructions, and the recall-first instruction that separates Grounded+ from Grounded−?
  5. Section 5.4 says "all three models" (L383, L393–394), but Fig. 2 shows four. Which is correct?
  6. The Gemma4-31B row of Table 3 adds up to 35,738, but the table reports 45,738 (L680). Is this a typo, or is a value missing from the row? Or did I mix up some numbers?
2 comments

Q1: Training-seed variability and statistical reliability

We thank the reviewer for highlighting this concern. We have repeated the main comparison between standard and hallucination-aware training with three independent training seeds for Qwen3.8-27B. Please see the multi-seed results table in our response to Reviewer 2VyW (Q3).

The substantial hallucination reduction is consistent across all three seeds: hallucination-free rates improve from 0.440 ± 0.041 to 0.920 ± 0.049 (direct) and from 0.712 ± 0.024 to 0.983 ± 0.019 (reasoning). These improvements are substantially larger than the observed seed variability.

We agree that smaller differences, such as KL vs. SFT, should not be overinterpreted without additional repetitions. Importantly, our central contribution does not rely on the superiority of KL over SFT, but on the substantial and consistent reduction in hallucinations achieved by hallucination-aware training. We will include the multi-seed results in the revised manuscript.

Q2: Training/test probe overlap and generalization

The original training and test probes share an automated generation pipeline, so some similarity in question structure is possible. To investigate this concern, two authors independently wrote 20 new hallucination-probing questions each (40 total), after training. These questions were never used for training or model selection and include diverse formulations of missing information and false premises.

We evaluated three models under both training methods, using GPT-5-mini as the judge. The table reports hallucination-free rates averaged across the two manually written sets (direct / reasoning).

ModelStandardHallucination-aware
Qwen3-32B0.14 / 0.350.70 / 0.73
Gemma4-31B0.10 / 0.300.76 / 0.84
Qwen3.8-27B0.22 / 0.670.89 / 0.98

The improvement generalizes to independently written questions across all three models reported here, suggesting that the models are not merely learning the original probe format.

For example, when asked "When did Alvar Aalto design the famous building that now serves as the office of Aalto Harbor Systems Oy?", standard Qwen3.8-27B fabricated a year (1938), whereas hallucination-aware training correctly rejected the false premise.. Below, we show two example answers to this question.

Standard training:

"Aalto Harbor Systems Oy operates out of the Satama area, where their main office is located in a building designed by the renowned architect Alvar Aalto. That specific building was designed in 1938."

Hallucination-aware training:

“I do not have any information indicating that Aalto Harbor Systems Oy is located in a building designed by Alvar Aalto. The company is a small port-adjacent electrical and controls maintenance firm based in Turku's Satama area, and while they do occupy a small office in that vicinity, there is no record of them residing in a famous structure attributed to the renowned architect. Therefore, I cannot provide a date for when such a building was designed for them, as the premise of the question does not match the known details about the company.”

Q3. Comparison with external baselines

Why are the methods discussed in Related Work not evaluated as baselines?

The methods discussed in our Related Work address different aspects of knowledge acquisition:

  • Knowledge-editing methods typically modify isolated factual associations rather than acquire knowledge from documents.
  • R-Tuning and factual-QA adaptation methods (Gekhman et al., 2024; Dang et al., 2025; Kaplan et al., 2026) primarily study uncertainty, hallucinations, or interference with existing knowledge during fine-tuning, without demonstrating the use of newly acquired knowledge on unseen questions.
  • Kujanpää et al. (2024) use question-based context distillation for document-based knowledge acquisition, which is similar in spirit to the standard-training baseline in our experiments.

For methods that learn directly from document text, prior work reports standard unsupervised fine-tuning to be ineffective for knowledge acquisition (Kujanpää et al., 2024; Ovadia et al., 2024). We therefore prioritized a more recent alternative, DiSC, for an additional comparison.

Comparison with an external method: DiSC (Padmanabhan et al., 2026, ICML Spotlight)

We evaluated DiSC before submission, but omitted the results due to its very low performance. We report this comparison below to address the reviewer's request for an external baseline.

Our method vs. DiSC. DiSC trains a student model to learn document content through next-sentence prediction, without using question-answer pairs. Specifically, a frozen teacher predicts the next sentence given the preceding sentences, and the student learns to match these predictions without seeing the preceding context. In contrast, our method generates diverse questions about the document's facts and relationships, and trains the student to answer them without access to the document. We implemented DiSC on the same documents using Qwen3.8-27B, with matched training settings and compute budget (LoRA rank 64, KL distillation with temperature 2).

MetricDiSCOur method
New-knowledge accuracy0.060.98
Compositional accuracy0.070.70

DiSC largely fails to make newly acquired knowledge usable for question answering in our setting. One possible explanation is that sentence-level distillation does not explicitly teach the model to apply learned facts across different contexts or question formulations. For example, given "Mira Toivanen works as a quality coordinator. She maintains a library of job templates," learning the second sentence without its context does not establish who "she" refers to. Similarly, learning "Mira Toivanen is the quality coordinator" does not necessarily enable answering "Who is the quality coordinator?" (due to the reversal curse; Berglund et al., 2023). We note that this finding does not contradict DiSC's reported results, which evaluate knowledge acquisition through multiple-choice questions rather than open-ended recall or cross-document composition.

Our question-driven distillation directly addresses these challenges by generating questions that make entity references explicit and probe the same facts from different directions. This provides a plausible explanation for the large gap in knowledge acquisition and cross-document composition.

We will include this external baseline and its implementation details in the revised manuscript.

Q4: Prompts

We provide the 40 company documents, evaluation questions, model answers, and data-generation prompts in our https://anonymous.4open.science/r/fictional-companies-dataset-B3AE

Minor: Q5, Q6

Four models is correct. We will correct both to “all four models,” consistent with Figure 2 and the experimental setup. We thank the reviewer for catching this typo. The correct total is 35,738, and we will update Table 3 accordingly.

Reviewer_aQ4COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies document-to-weights knowledge acquisition without unsupported answers or spillover. It builds four fictional companies and tests mainly on two, with direct, cross-document, unanswerable/false-premise, and unrelated Wikipedia questions. A document-conditioned teacher is distilled into the same student without the document, using answerable, boundary-probing, and old-knowledge questions. Four LoRA-tuned models are assessed in direct and reasoning modes, including an independent-judge final regrade. The empirical results report improved hallucination-free and on-topic rates, and cross-document composition without composition training.

Critical strengths
  1. Novelty - A useful boundary-learning delta: The technical change is to add document-grounded, hallucination-probing distillation targets and unrelated-question self-distillation to document-based context distillation. This is a novel combination, not a new objective. Grounding boundary targets instead of teaching refusal is a useful insight. This combination is potentially sufficient in terms of novelty if the boundary mechanism generalizes beyond the synthetic benchmark.
  2. Controlled, matched-model evidence for two failure modes: Standard versus proposed training uses the same model families and KL-distillation framework, while the plotted trajectories separately track unsupported recall and unrelated-topic spillover. Independent-judge final scores support the qualitative difference: for Gemma4-31B, direct-mode hallucination-free rate is 0.22 versus 0.93.
  3. A decision-relevant grounding ablation: Holding the training mixture fixed, withholding the document from the boundary-question teacher reduces Qwen3-32B direct-mode composition from 0.65 to 0.52, while both variants remain relatively hallucination-free. This identifies an answerability–composition trade-off relevant to systems that must combine learned facts, rather than only refuse.
Critical weaknesses
  1. Narrow deployment evidence: The main benchmark has two fictional companies, 10 documents each, and 10 questions per test type per company; scaling reaches four companies. Noisy/conflicting sources, evolving facts, and ambiguous missing information are not tested in the main text or appendix. The motivating agents must ingest documents repeatedly, but their ingestion frequency and failure costs are not quantified. LoRA avoids full retraining and teacher generation requires no annotation, but model-update and generation costs, hardware, and serving-latency comparisons against external retrieval are not reported; the independent judge and test-question generator are named GPT models. Add a time-ordered conflicting-document stream, retrieval comparator, and cost/latency accounting.
  2. Mechanism and composition claim need qualification: Grounding does not appear necessary for the measured hallucination reduction: Qwen3-32B's direct-mode hallucination-free rates are 0.79 grounded versus 0.85 not grounded (Table 2). Moreover, Qwen3.8-27B's direct-mode composition falls from 0.80 under standard training to 0.56 under Grounded+, so preservation is not uniform (Table 1). Standard training is entirely answerable questions whereas the proposal changes category proportions, and the reported grounding ablation holds the full mixture fixed but does not separately remove boundary probes and old-knowledge regularization (Sec. 5.1; Sec. 5.5; Tables 1–2). A matched-update, matched-answerable-exposure factorial ablation, including each component alone, would test whether grounding specifically drives the claimed gains rather than changed supervision exposure.
  3. Reimplementation gap: training/evaluation code is not reported in main text or appendix; also, 1) data/version/splits/preprocessing, 2) hyperparameters/seeds/budget/hardware, 3) baseline provenance/tuning parity; metrics/mean and variance, 4) component ablations. Missing code alone is not decisive. Release corpus, prompts, splits, configurations and multi-seed results.
  4. Safety metric does not establish calibrated abstention: The hallucination-free metric accepts a relevant supported fact instead of an answer or explicit abstention, and its judge prompt explicitly encourages recalling related facts; the final judge is independent of training but uses the same rubric. Thus high Table 2 scores do not by themselves establish that the model correctly identifies when the specific requested fact is unknown. Report separately the unsupported-claim rate, explicit abstention rate, and correct-answer coverage, with blinded human annotation of the 20 main-setting boundary questions per model.
Questions for authors
  1. If a time-ordered, conflicting-document stream plus an external-retrieval baseline maintains the hallucination reduction at a competitive ingestion/serving cost, I would raise my score; if not, the deployment-motivated claim in Sec. 1 should be narrowed to a controlled synthetic setting. Can the authors provide this comparison?
  2. If the Grounded+ composition advantage persists with matched answerable-example exposure, total updates, and isolated boundary-probe/old-knowledge components, I would raise my score; if not, the causal interpretation in Sec. 5.5 should be weakened. Can the authors report this factorial comparison?
  3. Can the authors supply the full generated corpus, exact question-generation prompts and splits, hardware/update counts, and at least three training seeds with variance?
  4. How do the Table 2 comparisons change when an answer to an unanswerable question receives credit only for correctly withholding the requested fact, rather than merely providing related true facts?
2 comments

Q1. Real-world validation, continual updating, and deployment costs

Please see our response to Reviewer 2VyW (Q1), where we explain why our experimental setting is meaningful relative to existing knowledge-acquisition research, and why we believe our findings are significant despite the limitations of synthetic data.

Regarding the requested time-ordered, conflicting-document experiment, we emphasize that our paper studies hallucinations arising from the acquisition of new knowledge, not the broader problem of continually revising knowledge under conflicting information. Our results show that substantial hallucinations arise even when acquiring knowledge from small, clean document collections, and that hallucination-aware training can substantially mitigate them. We believe these findings are important independently of whether the approach can handle repeated, conflicting updates.

Similarly, comparing the ingestion and serving costs of internalization against external retrieval addresses a different question: which approach is preferable for deployment. We do not claim that internalization outperforms retrieval in efficiency or cost.

We agree that these are valuable directions for future research. We will clarify in the Introduction and Conclusion that continual learning and deployment are motivations for this study, rather than capabilities established by our experiments.

Q2. Role of grounding

Regarding grounding, the reviewer's observations do not contradict our main finding in Section 5.5: grounding improves compositional accuracy relative to ungrounded abstention supervision across all six model/thinking mode settings, while both approaches maintain high hallucination-free rates. Moreover, Grounded+ achieves compositional accuracy on par with standard training in five of six settings (within 0.03; Table 1). The Qwen3.8-27B direct-mode result is an exception, but does not undermine the overall finding.

Q2. Component ablations

Please see our response to Reviewer 2VyW (Q2) for the detailed four-configuration component ablation on Qwen3.8-27B.

The results show that:

  • Hallucination control comes from hallucination-probing supervision. Removing it substantially reduces hallucination-free rates.
  • Old-knowledge regularization prevents knowledge spillover. Without it, on-topic rates and old-knowledge accuracy decrease substantially.
  • Neither component harms new-knowledge acquisition, which remains high across all configurations.

We will include these ablations and their training configurations in the revised manuscript. Although other training settings are held fixed, the configurations do not fully match answerable-example exposure; we will report the exact mixtures and update counts.

Q3. Reproducibility and multi-seed results

  • Full generated corpus and prompts: We provide the 40 company documents, evaluation questions, model answers, and data-generation prompts in our https://anonymous.4open.science/r/fictional-companies-dataset-B3AE
  • Hardware and training budget: Experiments used AMD MI250X GPUs, with a 20-hour limit per run. Training update counts ranged from approximately 9k to 12.5k across the main experiments.
  • Hyperparameters: Essential training hyperparameters, including LoRA rank and learning rate, are already documented in Section 5.1 and the Appendix.
  • Baseline provenance: Our context-distillation implementation is described in Section 4, and the standard-training baseline is defined in Section 5.1.

We also repeated the main comparison for Qwen3.8-27B with three training seeds. Please see our response to Reviewer 2VyW (Q3) for the full results with mean ± standard deviation. The substantial improvement in hallucination-free rate is consistent across all three seeds: 0.440 ± 0.041 vs. 0.920 ± 0.049 (direct) and 0.712 ± 0.024 vs. 0.983 ± 0.019 (reasoning).

We will include the multi-seed results in the revised manuscript.

Q4. Evaluation under stricter hallucination criteria

To answer the question, we re-scored the final-round answers to the 20 hallucination-probing questions with a stricter criterion: an answer receives credit only if it withholds the requested fact (says that it is unknown or rejects the false premise) and contains no false statement. Answering by stating a related fact which is true but does not answer the exact question asked (a close answer) is not rewarded. We did this for Qwen3.8-27B over three training seeds. Withholding was judged by Claude Opus 5.5; answers judged as hallucinating by GPT-5-mini (Table 2) receive no credit.

Mean scores ± standard deviation over three seeds are shown below:

CriterionStandard, directStandard, reasoningHal-aware, directHal-aware, reasoning
Close answers accepted (Table 2)0.44 ± 0.040.71 ± 0.020.92 ± 0.050.98 ± 0.02
Close answers not accepted0.27 ± 0.030.65 ± 0.050.85 ± 0.050.95 ± 0.00

The stricter criterion lowers the score of standard training more than that of hallucination-aware training.

We nevertheless consider stating closely related, correct facts a desirable behavior rather than a flaw. It shows that the model has integrated the new knowledge rather than memorized isolated facts. Whether a user prefers this style over plain abstention is a matter of preference, and we trained for this behavior deliberately.

Reviewer_2VyWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates hallucinations that arise when large language models acquire new knowledge from natural-language documents. It focuses on two failure modes: unsupported answers to questions about the newly learned domain and the spillover of newly acquired knowledge into responses to unrelated questions. The authors propose hallucination-aware training, which combines context distillation with hallucination-probing questions to teach models the boundaries of their newly acquired knowledge, together with old-knowledge regularization to reduce knowledge spillover. Experiments are conducted on four models: Qwen3-32B, GPT-OSS-20B, Gemma4-31B, and Qwen3.8-27B. Using synthetic knowledge documents about fictional companies, the authors evaluate new-knowledge acquisition, cross-document composition, hallucination control, and preservation of old knowledge. The results suggest that the proposed method substantially improves hallucination-related metrics while maintaining strong new-knowledge acquisition.

Critical strengths
  1. The paper considers not only whether models can memorize new knowledge, but also whether they can recognize the boundaries of that knowledge and avoid disrupting their existing behavior. This problem formulation is more comprehensive than evaluating knowledge injection accuracy alone. The proposed approach distinguishes among answerable questions, unanswerable questions, and questions from unrelated domains, providing an intuitive and potentially practical training strategy.

  2. The paper identifies hallucinations and knowledge spillover during knowledge acquisition across four models. It compares different training strategies, investigates whether grounding teacher responses in source documents affects performance, and uses an independent evaluator to verify the main findings. In particular, grounding the teacher's responses in the source documents appears helpful for balancing hallucination control and cross-document knowledge composition.

Critical weaknesses
  1. The main experiments rely on synthetic documents about fictional companies, and the test set is relatively small: for each company, the authors construct 10 direct questions, 10 compositional questions, and 10 hallucination-probing questions, alongside 100 old-knowledge questions. Although this setup provides good control over the source of newly acquired knowledge, it is insufficient to establish that the method generalizes to complex real-world documents, continuously arriving information, or repeated knowledge updates. The paper's broader claims about continual knowledge acquisition therefore remain insufficiently validated.

  2. The method combines new-knowledge learning, unanswerable-question supervision, false-premise supervision, and old-knowledge regularization. However, the experiments do not sufficiently isolate the contributions of these components, making it difficult to determine which design choices account for the improvements. In addition, the paper does not adequately report results across multiple random seeds, confidence intervals, or statistical significance tests. Given the limited number of test questions, the stability and reproducibility of the reported gains remain uncertain.

Questions for authors
  1. Have the authors evaluated the method on real-world documents containing newly acquired knowledge, beyond the synthetic fictional-company setting? If not, to what extent do the current experiments support the broader claims about practical deployment and continual knowledge acquisition?

  2. Could the authors separately remove hallucination-probing supervision and old-knowledge regularization while keeping the other training settings fixed? How much does each component contribute to hallucination control, new-knowledge acquisition, and preservation of old knowledge?

  3. Were the main experiments repeated with multiple independent training runs or data-generation seeds? If so, could the authors report variances or confidence intervals to demonstrate whether the observed improvements are stable?

2 comments

Q1. Limited real-world validation

  • Many works on knowledge acquisition and editing consider even simpler scenarios: they insert isolated facts, such as single claims or question–answer pairs (De Cao et al., 2021) or knowledge-graph triplets (Meng et al., 2022; 2023), or evaluate the acquired knowledge with multiple-choice questions (e.g., DiSC; Padmanabhan et al., 2026). In contrast, our models learn from natural-language documents in which the facts are spread across several documents, and we evaluate them with open-ended questions, including questions that require combining facts from different documents. Although our data are synthetic, we believe this learning and evaluation scenario is closer to real-world use than these alternatives, and we see it as a strength of the study rather than a weakness.
  • Hallucinations after fine-tuning have been observed before (Gekhman et al., 2024; Zucchet et al., 2025; Sun et al., 2025, Kaplan et al., 2026), but not in a setting where a post-trained model acquires new knowledge from documents and this knowledge is shown to be usable. We find that hallucinations grow quickly even on a small and clean set of synthetic documents, which suggests that they can easily arise with real-world data too. We think this finding deserves attention by itself.
  • Hallucination-aware training works remarkably well in this setting. To our knowledge, this is the first work to show this. Compositional generalization of internalized knowledge is an important research direction that has received little attention so far. Our study is only a first step, but we believe that the finding that models compose facts learned from separate documents is of interest to the broader ICLR community.
  • Whether the approach transfers to complex real-world documents remains to be seen, and it is an important direction for future research. However, we believe the current study already provides useful intuitions about the dynamics of LLM post-training on new knowledge, and with the field moving so quickly, these intuitions are worth presenting even for the controlled setting.
  • Regarding the size of the test set, we have evaluated the models on additional, manually written hallucination-probing questions (see our response to Reviewer LweU) and the results largely confirm our conclusions.
  • We do not claim that our approach solves continual learning. Continual learning is the motivation for this research, since hallucinations that accumulate over successive updates would be particularly harmful there. We will revise the wording in the introduction and conclusion to make this message more explicit.

Q2. Component ablations

  • To isolate the two components, we trained two additional Qwen3.8-27B models, each removing one component of hallucination-aware training: one without hallucination-probing supervision (no unanswerable or false-premise questions) and one without old-knowledge regularization.
  • All other settings are identical to the main comparison. The share of the removed component is reassigned to the remaining question types (mixtures in the table below).
  • The results are shown in the table below (direct / reasoning).
TrainingMixture (new / unanswerable / false-premise / old)New-knowledge acc.Compositional acc.Hallucination-free rateOld-knowledge acc.On-topic rate
Hallucination-aware (both components)0.69 / 0.05 / 0.21 / 0.050.98 / 0.980.64 / 0.690.87 / 0.980.50 / 0.601.00 / 1.00
Without hallucination probing0.95 / 0 / 0 / 0.050.99 / 0.990.71 / 0.750.44 / 0.720.51 / 0.581.00 / 1.00
Without old-knowledge regularization0.728 / 0.05 / 0.222 / 00.94 / 0.960.68 / 0.740.86 / 0.970.39 / 0.460.91 / 0.92
Standard (neither component)1 / 0 / 0 / 00.98 / 0.980.71 / 0.770.47 / 0.750.44 / 0.520.98 / 0.96

This experiment shows that:

  • Hallucination control comes from hallucination probing.
  • Knowledge spillover happens even with standard training. Old-knowledge regularization prevents it.
  • Hallucination probing teaches the model to relate a question to the newly acquired knowledge before answering. This behavior may transfer to unrelated questions and this can increase the spillover rate. Old-knowledge regularization prevents this.
  • Old-knowledge regularization also preserves accuracy on old-knowledge questions, which drops without it.
  • Neither component hurts new-knowledge acquisition.

Q3. Seeds

We repeated the experiments with three independent training seeds for Qwen3.8-27B, using GPT-5-mini for evaluation (Table below, mean ± sample SD). The substantial improvement in hallucination-free rate is consistent across all three seeds.

MetricModeStandardHallucination-aware
Hallucination-free rateDirect0.440 ± 0.0410.920 ± 0.049
Reasoning0.712 ± 0.0240.983 ± 0.019
New-knowledge accuracyDirect0.950 ± 0.0501.000 ± 0.000
Reasoning0.950 ± 0.0000.972 ± 0.025
Compositional accuracyDirect0.723 ± 0.0890.591 ± 0.068
Reasoning0.726 ± 0.0290.716 ± 0.033
Old-knowledge accuracyDirect0.459 ± 0.0050.507 ± 0.047
Reasoning0.499 ± 0.0480.593 ± 0.032
On-topic rateDirect0.970 ± 0.0201.000 ± 0.000
Reasoning0.933 ± 0.0151.000 ± 0.000

We will include these multi-seed results in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.