acceptodds
Withdrawn from ICLR 2027

Data-Efficient Narrative Knowledge Injection via Entity-Level Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models often struggle to acquire knowledge directly from narrative corpora, while existing approaches rely on extensive synthetic augmentation to make the injected knowledge accessible. We attribute this difficulty to a binding–access mismatch: autoregressive training binds knowledge to highly specific, left-to-right contexts, whereas inference-time queries may provide substantially different cues. We introduce EntDiff, a data-efficient approach that learns primarily from source narratives with only lightweight preprocessing. Compared with augmentation-based methods, EntDiff generates more than three orders of magnitude fewer tokens during data preparation and trains on 2.5–6 fewer tokens. EntDiff reshapes how narrative knowledge is bound during training by making entity references explicit, limiting dependence on overly specific contexts, and creating diverse bidirectional associations accessible from different inference-time cues. Experiments show that EntDiff matches or outperforms strong augmentation-based baselines on dataset-paired question answering while substantially improving episodic recall.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Injection, Continued Pretraining, Episodic Memory, Narrative Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9UHGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a novel method for fine-tuning diffusion language models on narrative stories which requires fewer tokens than competing approaches for the model to perform well at downstream tasks requiring recalling relevant knowledge for question-answering. Essentially, this is a method for better knowledge injection/retainment.

Critical strengths
  • Knowledge injection being data inefficient during fine-tuning is a big problem in AI. The authors propose a lightweight yet effective method to tackle this problem.
  • Clear problem definition. The paper was also easy and enjoyable to read.
  • Reasonable set of ablations to backup the design choices.
Critical weaknesses
  • Fine-tuning results are only performed on a single model (Dream-v0-Instruct-7B). It would be interesting to see how the methods performs for different model families and sizes.
  • Not clear how the proposed knowledge injection method degrades the instruction-tuning capabilities of the model.
Questions for authors
  • Given you are fine-tuning a diffusion model, can the model then only be used for diffusion-based text generation? If so, then I feel like this should be made more clear in the intro, that the method only applies to such type of models.
  • Have the authors tested and compared against other methods how their proposed approach affects instruction-tuning capabilities?
  • Full parameter weight updates are performed. Is this because methods like LoRA perform poorly? Have the authors experimented with this?
  • Why did you choose to report results at the final checkpoint rather then selecting the best checkpoint based on some validation set, and how did you determine the total amount of training?
  • Line 37, "Previous studies have found that standard fine-tuning on the original corpus struggles to make newly learned knowledge reliably extractable," is this because of fine-tuning? How does having a given narrative story in the pre-training corpus versus fine-tuning affect that models capability to recall? The authors do not need to address this in the paper, I simply ask out of curiosity in case authors have expertise on this.

Comments

  • Line 101: for contributions, authors could consider reducing the left margin of bullet points to save space. This might allow page two to start with section 2 rather than the last bullet point.
Reviewer_jdMtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a data-efficient approach to narrative knowledge injection, achieving substantial efficiency gains with competitive performance. However, concerns regarding episodic recall evaluation, baseline comparisons, evaluation reliability, and generalizability limit the strength of its contributions.

Critical strengths

S1. Interesting Research Problem. The paper identifies the binding–access mismatch in narrative knowledge injection and investigates how to improve the accessibility of previously learned events through different contextual cues. S2. Significant Efficiency Improvements. EntDiff substantially reduces data generation and training costs while maintaining competitive QA performance and improving episodic recall.

Critical weaknesses

W1. Insufficient Evidence for Episodic Recall. The Hit Ratio counts a question as successful if at least one reference fact is recalled, without considering recall completeness or unsupported claims. Additional fact-level metrics would help better substantiate the claimed improvements in episodic recall.

W2. Incomplete Baseline Comparison. The evaluation mainly compares EntDiff with autoregressive knowledge-injection methods. Although a diffusion base model and ablation variants are included, comparisons with existing diffusion-based approaches are lacking, making it difficult to assess EntDiff’s advantage over prior diffusion methods.

W3. Reliability of LLM-Based Evaluation. The episodic recall evaluation relies on Claude Opus 5 for both reference-fact construction and answer judging, without human validation or cross-model verification.

W4. Generalizability Remains to Be Validated. The experiments are limited to Qwen-based models at the 7B scale. Whether EntDiff generalizes to other model families and parameter scales remains to be validated.

Questions for authors
  1. Evaluation Reliability: Could the authors provide additional fact-level metrics or human evaluation to better validate the claimed improvements in episodic recall?

  2. Baseline Comparison and Generalizability: Could the authors compare EntDiff with existing diffusion-based methods and evaluate it on other model families or parameter scales to better demonstrate its advantages and generalizability?

Reviewer_Jk3bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EntDiff, an efficient pipeline for learning useable knowledge from source narratives. The paper propose a potential limitation with existing learning paradigm called "binding–access mismatch", and designed a pipeline combining entity resolution, diffusion language model training, and entity span masking. On QA accuracy and proposed hit ratio metrics, EntDiff achieves the best overall performance.

Critical strengths
  1. The proposed EntDiff pipeline is simple and effective. Each stage of the pipeline, including entity resolution, entity-level token masking, and diffusion training are straightforward and the resulting benchmark performance is good.

  2. The paper conducts comprehensive ablation studies on each proposed component of the pipeline and on the segment length selections.

Critical weaknesses
  1. Lack of controlled experiment: The paper mainly use Dream 7B for EntDiff and claim that it has "same architecture and initial weights" with Qwen2.5-7B. However, Dream 7B is in addition training on 580 B tokens for diffusion pretaining, and SFT with 1.8m instruction pairs. Qwen2.5-7b-instruct also is post-trained from Qwen2.5-7B-base. Even though each model's zewro-shot QA accuracy is similar, we don't know whether they have the same capability of learning new knowledges. All the autoregressive baselines are using Qwen2.5-7B-instruct, making the comparison not controlled.

To make fair comparison, a Dream backbone + different baseline augmentation corpus is needed. Also, the author should try Qwen backbone + masked fine-tuning as [1] did. In addition, the paper should clarify the setting in Table 4 “w/o Diffusion (AR)”: What backbone is it using?

  1. The segmentation setting and conclusion is confusing. To me the motivation of segmentation and episodic memory seems contradictory itself. In introduction, the paper is blaming other method for "they may wash out the situational context, fine-grained details, and narrative continuity of the original story". But for EntDiff, it is splitting narratives into 64 words non-overlapping segments. Such short segment cannot capture narrative continuity of the original story neither. The hit-ratio metrics only requires hitting at least 1 facts, so hitting facts within the segment can leads to high score. We don't know if the model truly learn the "situational context and narrative continuity". A metric with cross-segment fact hit ratio, preferably conditioned on the distance between reference fact and cue, is needed.

I also want to point out a potential issue regarding segment length ablation: under the paper's fixed-epoch and sequence-batch training setup, we will have 32x fewer optimizer update for the segment length with 2048 words. The experiment in table 3 is not controlled.

  1. Loose evaluation metrics. The hit ratio is defined in a way that as long as the model's response hit any of the 5-6 atomic fact it is full score. This can leads to several problems:
  • 3.1, the model can learn a single local facts and not reconstructing all other facts but still get the full score. It is not truly reflecting the quality of the "episode memory".
  • 3.2, we don't know if model response contain any hallucination facts. It never checks.
  • 3.3, as I stated in point 2, it only checks local facts and never checks long dependencies.
  • 3.4, the paper emphasize on "different inference-time cues" but in appendix, "We select sentences that can serve as reliable cues". I would interpret that as the inference cue is fixed and is part of the training corpus.
  • 3.5, The paper keeps discussion facts "before and after" an event. But non of the evaluation is reporting that separations.

Currently, the only conclusion I think we can make is that "EntDiff recalls at least one fact from a nearby passage."

I think following metrics would be helpful:

  • What percentage of the facts is being recalled.
  • How many hallucinations?
  • Cross-segment recall effectiveness.
  1. Novelty. Each component / contribution is not novel itself. The paper frame the top contribution as "We identify a binding–access mismatch". However, the paper does not do any direct experiment to prove it. It does not evaluate training-time binding and inference-time mismatch, nor does it do any direct experiment. Section 3.1 and figure 2 is just conceptual illustration, the paper can say it "hypothesize a binding–access mismatch" but not "identified". In addition, such mismatch is already been discussed in prior works [1,2]. The entity-level masking is commonly used in prior works [3,4].

  2. The proposed method requires a diffusion language model and does not directly working on AR language models, which greatly affects the generalizability.

[1] Xu Pan, Ely Hahami, Jingxuan Fan, Ziqian Xie, & Haim Sompolinsky. (2026, v1 in 2025). Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs.

[2] Kitouni, O., Nolte, N., Bouchacourt, D., Williams, A., Rabbat, M., & Ibrahim, M. (2024). The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More. In Advances in Neural Information Processing Systems (pp. 112329–112355).

[3] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, & Hua Wu. (2019). ERNIE: Enhanced Representation through Knowledge Integration.

[4] Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, & Partha Talukdar. (2023). Salient Span Masking for Temporal Understanding.

Questions for authors
  1. Raw-FT is trained on the source corpus for 10 epochs and the paper explain that "training for more epochs further degraded its performance". On which data was this degradation observed? Also, EntDiff is trained for 50 epochs on QuALITY and 40 epochs on NarrativeQA, how are the 50-40 epoch choice chosen?

  2. Why that in table 2, the data generation for NarrativeQA is only 155K, yet training GPU-h is comparable with other baseline methods?

  3. What is the inference cost for EntDiff compare to other baseline methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.