- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The paper proposes KnowContext, a set of twelve structured rewriting strategies for training language models on updated or counterfactual facts. It evaluates direct and inverse recall and several downstream tasks across seven domains and three models. KnowContext improves the reported six-task average over DeepKI by roughly 20–25 points.
Critical strengths
The task-by-task evaluation shows why direct fact recall alone is an incomplete measure of a knowledge update. Matching the main rewriting baselines on training-sample count makes the large observed gains worth investigating.
Critical weaknesses
The paper frames the method as a targeted knowledge update but does not measure preservation of unrelated facts or general capabilities, as its limitations section acknowledges. Without that measurement, high scores on updated facts do not establish whether the overall update is beneficial.
Both training and evaluation material are synthetic. The paper checks exact duplication, reserved-anchor overlap, and some leave-one-format-out settings, but the latter cover only selected models and domains. Three of the six task families also depend on an LLM judge. These checks do not yet establish that the large average gains transfer to independently written questions or reliably judged answers.
Questions for authors
What happens to unrelated factual recall and general-task performance after KnowContext training? Can the authors also report results on independently authored evaluation questions, with human agreement for the open-ended answer judgments? Either result could change my recommendation.
6 comments
Ans to W2a, W2b and Q2
W2a & Q2 – Transfer to independently sourced evaluation questions
Ans to W2a & Q2
We evaluate DeepKI and KnowContext on the public MQuAKE-Remastered dataset. After filtering for single-edit cases with valid reasoning chains and removing duplicate edits, we sample 60 atomic facts and retain their 180 official questions for evaluation only. We rewrite each fact with both methods using the same budget of 150 rewrites, then train following Sec. 4.1 and Appendix B.1. The results are summarized below.
Official QA uses the public MQuAKE questions. AVG averages only our six evaluation tasks (DQA–TOL), excluding Official QA. Parentheses give KnowContext − DeepKI in points for Official QA and AVG. Scores are percentages.
| Method | Official QA | DQA | IQA | MCQ | MHP | SCE | TOL | AVG |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | ||||||||
| DeepKI | 15.56 | 96.67 | 47.78 | 23.33 | 38.33 | 72.78 | 41.11 | 53.33 |
| KnowContext | 31.11 (+15.56) | 98.89 | 60.00 | 69.44 | 62.78 | 90.56 | 81.11 | 77.13 (+23.80) |
| Llama-3.1-8B | ||||||||
| DeepKI | 12.78 | 99.44 | 48.33 | 43.06 | 28.89 | 66.67 | 7.78 | 49.03 |
| KnowContext | 34.44 (+21.67) | 97.78 | 63.33 | 67.50 | 66.67 | 82.22 | 78.33 | 75.97 (+26.94) |
| Qwen3-14B | ||||||||
| DeepKI | 18.89 | 98.33 | 56.67 | 35.83 | 43.33 | 73.89 | 44.44 | 58.75 |
| KnowContext | 48.33 (+29.44) | 100.00 | 55.00 | 80.83 | 69.44 | 92.78 | 79.44 | 79.58 (+20.83) |
KnowContext shows clear improvements on both the official questions (+15.56–29.44 points) and our evaluation questions (+20.83–26.94 points in AVG) across all three backbones. Its advantage therefore extends to evaluation questions sourced independently of our pipeline.
W2b – Broader leave-one-format-out controls
Ans to W2b
For the concern about overlap between training and evaluation formats, Table 7 already reports Qwen3-8B on GEO/BIO and Llama-3.1-8B on GEO. We expanded the controls to all seven domains on Qwen3-8B and GEO on all three backbones, including −Inverse conditions absent from the original BIO and Llama experiments. We removed Inverse or Discrimination rewrites, matched Full's training exposure, and kept the evaluation questions and scoring fixed.
Parentheses give leave-out − Full in points for IQA, MCQ, and AVG, calculated before rounding.
| Domain | IQA Full → −Inverse (%) | MCQ Full → −Discrimination (%) | AVG Full / −Inverse / −Discrimination (%) |
|---|---|---|---|
| Qwen3-8B | |||
| GEO | 84.92 → 85.47 (+0.56) | 78.61 → 70.00 (−8.61) | 88.82 / 84.79 (−4.03) / 86.37 (−2.45) |
| BIO | 92.18 → 89.94 (−2.23) | 86.42 → 76.59 (−9.83) | 84.71 / 78.14 (−6.57) / 74.42 (−10.29) |
| Brand | 59.44 → 39.44 (−20.00) | 72.33 → 76.42 (+4.09) | 65.78 / 60.05 (−5.73) / 70.93 (+5.15) |
| Creative | 92.78 → 92.78 (0.00) | 35.75 → 82.12 (+46.37) | 76.89 / 81.26 (+4.37) / 83.86 (+6.97) |
| Game | 85.00 → 82.78 (−2.22) | 50.00 → 42.35 (−7.65) | 75.38 / 71.25 (−4.14) / 75.52 (+0.14) |
| Materials | 93.89 → 95.56 (+1.67) | 84.44 → 73.89 (−10.56) | 80.93 / 79.07 (−1.85) / 79.81 (−1.11) |
| History | 80.45 → 74.86 (−5.59) | 73.28 → 34.77 (−38.51) | 77.28 / 71.45 (−5.83) / 68.91 (−8.37) |
| Llama-3.1-8B | |||
| GEO | 79.89 → 74.86 (−5.03) | 64.72 → 73.89 (+9.17) | 79.36 / 79.77 (+0.41) / 77.36 (−2.00) |
| Qwen3-14B | |||
| GEO | 83.24 → 84.92 (+1.68) | 83.33 → 68.33 (−15.00) | 85.63 / 82.98 (−2.64) / 82.38 (−3.24) |
All 18 leave-out conditions retain 60.05–86.37% AVG and remain 5.87–32.96 points above DeepKI in Table 5. This supports the paper's conclusion that direct format matching does not explain KnowContext's overall advantage. Individual task effects remain domain-dependent, including a 20.00-point IQA drop on Brand and a 38.51-point MCQ drop on History.
Ans to W2c
W2c – Human agreement and judge sensitivity
Ans to W2c
For the concern about reliable judgments of open-ended answers, our main evaluation uses GPT-5.5 at temperature 0 for MHP, SCE, and TOL. The scoring uses simple binary criteria that check whether the answer applies the updated fact and completes the requested task.
1) Agreement among human judges and with GPT-5.5
We sampled 90 GEO questions per task from MHP, SCE, and TOL and kept the DeepKI and KnowContext answers to each question, for 540 answers in total. Three human judges independently scored each answer using the same task rubrics as GPT-5.5.
| Comparison | Agreement (%) | Fleiss’ κ |
|---|---|---|
| Human majority vote vs. GPT-5.5 | 91.30 | 0.771 |
| Unanimous agreement among all three human judges | 95.74 | 0.921 |
| Unanimous agreement among all three human judges and GPT-5.5 | 90.00 | 0.852 |
Agreement is the match rate for human majority vote versus GPT-5.5, and the unanimity rate for the three- and four-judge comparisons.
The strong agreement with the human majority supports GPT-5.5's reliability under our simple binary rubrics, while the high agreement among human judges shows that the criteria can be applied consistently.
2) Sensitivity to the automated judge
We rescored saved MHP, SCE, and TOL answers from the Qwen3-8B matched-token DeepKI and KnowContext runs on GEO and BIO with DeepSeek V4 Pro at temperature 0 with thinking disabled. We compared its decisions with those of GPT-5.5, using the same questions, answers, references, and task rubrics. The accuracy columns below use the DeepSeek judge and average the three task accuracies.
| Domain | DeepKI (%) | KnowContext (%) | Δ under DeepSeek (points) | Δ under GPT-5.5 (points) | Decision agreement (%) |
|---|---|---|---|---|---|
| GEO | 57.15 | 80.34 | +23.19 | +24.30 | 91.47 |
| BIO | 35.11 | 81.15 | +46.04 | +46.00 | 92.16 |
KnowContext retains its lead under both supplementary judges. Its margin changes by at most 1.11 points across GEO and BIO.
References
[1] Ibrahim et al. (2024). Simple and Scalable Strategies to Continually Pre-train Large Language Models. Transactions on Machine Learning Research.
[2] Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS.
[3] Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding. ICLR’21.
[4] Clark et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.
[5] Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? ACL’19.
[6] Rolnick et al. (2019). Experience Replay for Continual Learning. NeurIPS’19.
[7] Kotha et al. (2024). Understanding Catastrophic Forgetting in Language Models via Implicit Inference. ICLR’24.
[8] Diao et al. (2026). Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting. Findings of EMNLP’26.
Ans to W1 and Q1
We appreciate the reviewer's constructive suggestions on preserving pretrained knowledge and independently validating generalization and evaluation reliability.
Due to limited time and GPU resources, we focused our additional experiments on GEO and BIO.
W1 & Q1 – Preservation of unrelated facts and general capabilities
Ans to W1 & Q1
Our main experiments isolate knowledge-injection effectiveness without interventions against catastrophic forgetting [2]. Fine-tuning can bias models toward the update distribution and reduce pretrained capabilities [7]; conflicting supervision can further contribute to forgetting [8].
In practice, forgetting can be mitigated by mixing a common corpus into the update data [1] or replaying data from earlier training stages [6]. We additionally mix WikiText-2 into the training data with a 10% token budget on both 8B backbones and report the results below.
We evaluate preservation of:
1) Retention of unrelated facts
Parentheses give Δ from Base, except for GEO AVG, which uses the same method without mixing as its reference.
2) General capabilities
We make the following observations:
Common-corpus mixing mitigates forgetting of both unrelated facts and general capabilities. After mixing, KnowContext's gaps from Base are 2.64–9.46 points on unrelated facts and 3.22–5.60 points averaged across the three general benchmarks. The 10% mixture is exploratory and not tuned for the best results.
Adding a common corpus causes only small decreases in targeted knowledge-update accuracy. KnowContext's GEO AVG decreases by only 0.60–1.43 points, while remaining 17.40–22.73 points above DeepKI.