acceptodds
Under review as a conference paper at ICLR 2027

Story-Gradient Editing Can Misattribute Newly Learned Real Facts

Abstract

Language models learn from factual records and fiction. They need to remember both without confusing which events are real. Epistemic Goggles is an existing method that edits training gradients so that models remember learned material as fictional. We test whether this framing reaches new real facts learned alongside the stories. Matched Qwen3-8B runs share initialization, documents, and update count, and only story updates use the editor. Edited runs retain real awards but report them as fictional even when ordinary training learns their correct status. Synthetic events with varied relations and persons show the same separation between content and attribution. Correct source reports also fail to ensure consistent judgments and applications of the same events. Source supervision improves some answers without making these behaviors agree. The attribution effect also reaches events absent from the edited documents. At half amplitude, both tested synthetic donor updates change correct real-award judgments and retain award content. Three spectrum-preserving random directions retain the original judgments. These results show why selective learning requires more than retaining prior knowledge or reproducing a source answer. Newly learned facts must acquire their own status and support its use across questions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
continual learning, knowledge acquisition, source attribution, fiction versus fact, gradient editing, LoRA, parameter transfer, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PfKZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to use Epistemic Goggles (a gradient editing approach proposed by prior work) to help models distinguish real or fictional facts. The writing is terrible so much of the content basically can't be understood, especially with respect to the contributions of this work, experimental setups and results.

Critical strengths

none

Critical weaknesses

Much of the content in this paper is badly written. Technical contributions are unclear. Experiments are presented unclearly.

Questions for authors

none

Reviewer_FoZWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether Epistemic Goggles, a pretrained gradient editor designed to make models remember story content as fictional, affects newly learned real facts. Using matched LoRA training on Qwen3-8B, the authors alternate real-event and fictional-story documents and edit only the story gradients. In one real-award setting, ordinary training correctly recalls and attributes 7/8 awards, while Goggles + clip retains all eight but labels them fictional. Further experiments examine inconsistencies across recall, claim judgments, and applications, as well as the effects of transferring edited updates to models trained on other events. The paper argues that selective learning must preserve both factual content and its source status.

Critical strengths
  • The matched ordinary-versus-edited training runs control initialization, documents, and update counts. The real-only baseline helps show that the new awards can be learned with correct attribution under the same exposure.
  • Separating content recall, source reports, claim judgments, and applications is useful. The examples of contradictory answers to the same event make the measurement issue concrete.
  • The parameter-transfer experiments include dose variation and spectrum-preserving random directions. The half-strength results, which change source judgments while retaining award names, are an interesting observation.
Critical weaknesses
  • The research setting lacks a convincing motivation. Since the pipeline already identifies fictional documents, the paper should explain when retaining them in model parameters is needed, rather than filtering them or storing them for retrieval. It also does not establish why gradient editing is preferable to explicit source supervision. The Introduction gives no concrete application of this setup.
  • The Introduction is too brief and cites only Epistemic Goggles. It moves directly from models learning facts and fiction to this editor's failure mode, without explaining the wider selective-learning problem or limitations of existing alternatives. Although some related studies appear later, the motivation does not establish a clear research gap or explain why this specific failure matters.
  • The contribution mainly documents a failure of one existing method. While the misattribution effect is clear, its mechanism remains uncertain: real and story updates share Adam state, and the parameter-transfer experiments do not isolate how an unrelated fact acquires fictional status. It remains unclear whether this is a Goggles-specific side effect or a broader selective-learning phenomenon.
  • The experimental scope is narrow. All main experiments use Qwen3-8B and small LoRA-trained event pairs; the real-award cohort includes just eight people, with awards paired against fictional inventions. Synthetic counterbalancing and same-relation controls partly address this, but broader models, editors, and realistic mixed-document training would be needed to support the generality of the findings.
  • The findings are not integrated into a clear explanation. Inconsistent judgments across question forms also occur after ordinary or real-only training, weakening their connection to Goggles. The strongest transfer result uses only two synthetic donors and seven initially correct awards, leaving open whether the effects reflect a distinctive attribution mechanism or generic interference from parameter updates.
Questions for authors
  • What concrete application requires learning known-fictional stories into model parameters? What would be lost by filtering them for factual tasks or keeping them as source-labeled external documents?
  • What does Epistemic Goggles offer over explicit source supervision in this setting, especially given the source-aware KL results on synthetic events?
  • Can the authors separate the direct effect of story-gradient editing from interference through shared optimizer state, and show whether a comparable error occurs with other editors?
  • How often does the misattribution occur on larger, more realistic mixed-source training data and across model families? Can the transfer finding be replicated with more donors and target facts?
Reviewer_2P6JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Epistemic Goggle is a quite recent method that was built to train models into believing that a span of text is e.g. fictional. Motivated by phenomena like negation neglect, where ordinary training tends to ignore labels such as "this is false" or "this is fiction" when they appear in the training data, Goggles allow to "mark" entire spans of text as e.g. fiction during training. This paper is a sequel or an analysis of Goggles that wonders what happens to real facts that are related to the fictional story, if they are learned at the same time.

In particular, it trains a small LLM on pairs of events about the same person: one real, and one fictional, opposing a Goggles run to a classic training run (A Goggle run uses Goggle editing only for the fictional story, but ordinary training for the real one). It finds that unlike ordinary training which gets it right, the real fact is labeled fiction by the Goggle trained model, even though Goggles never touched it. The paper further reports that even after ordinary training, whether a model will consider an event as real depends on the way the question is asked. General, this work motivates the fact that a method that wants to label a span as fictional, must also be tested on whether related real facts keep their "real" status.

Critical strengths
  • The Goggle problem is interesting: Making models distinct fiction from reality is timely and important. And so is the problem of undesirable transfer of learning properties which is reminiscent of other phenomena (emergent misalignment, subliminal learning etc.) This paper presents somehow another instance of these problems.
Critical weaknesses
  • Writing clarity: The writing does not feel natural throughout the paper. Terms are used without being defined, making it hard for a human reader to connect the dots. The abstract is unreadable, talks already about "awards" as if they were already know to the reader, while this comes later in the paper. The paper also uses different terms to signify the same thing which is counterproductive for readability (e.g. factual records, fictional story, story.. Note that a story can be true, it's a narrative about real or imagined events...). The difference between the ordinary training and the Goggle is not easily "parsable" from the text, and needs an investigation to get right (assuming i'm correct that in what is called the Goggle run, there's an ordinary training step on the real announcements, only Section 3.1, paragraph 2 was the clearest about it, the first mentions were vague). In general the paper felt as if it was written for someone who knew already all the work. This is by itself enough of a show stopper. Examples of awkward expressions and jargon defined before use: "ordinary-gradient" web material (data itself is not "ordinary-gradient"..), source-cued recall, archive-established, supplied-claim, donor...

  • Narrow scope and over-dependence on recent non-verified method: The title makes a general claim about "story-editing" as if it was an established known method but the entire work is heavily dependent on Epistemic Goggles, a non peer-reviewed arxiv submission that was out around 3 months before the ICLR submission. This is per se not a reason to reject, but heavily diminishes the motivation of the work and the breadth of the findings.

  • Further limited impact: The results of the paper point rather to the ill-design of Goggles, and less to something fundamental about LLMs (although I believe some links might exist to many papers that show habits that transfer beyond the current training run, including emergent misalignment, subliminal learning and so on). Besides What's the advantage of all this Goggling compared to simple rewriting? LLMs are cheaper and cheaper and synthetic data generation is not only routine, but what fostered the development of LLMs. What's wrong with just doing this? It would have been more important to focus the study of this "contagion" phenomenon in the broader scope of LLM training instead of the narrow Goggling scenario.

Questions for authors

Adding Goggle weight changes to another model could have been better motivated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.