- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors identify context-induced difficulty underestimation as the phenomenon in which later facts in a passage receive lower loss due to the preceding context. The authors identify this issue as one of the core components leading to the failure mode of context reliance. To counter this, the authors propose FOVEATED, a method that constructs focused views of individual sentences through randomly shifting position embeddings.
Critical strengths
-
The paper is overall well written with a clear structure that makes it easy to follow. The authors clearly identify a problem, define it, and then use it to motivate their method.
-
The proposed method is clearly motivated and backed by a theoretical analysis. Overall, the method is a simple and elegant fix to the identified problem.
-
The empirical analysis is thorough, covering two model families, three benchmarks, and five editing methods.
Critical weaknesses
-
While the problem is presented clearly, the intuitive explanation is not completely substantiated by the empirical assessment. While Figure 1 shows a consistent trend of lower losses for later position IDs, it is not clear whether the lower losses are driven by the previous context. Intuitively speaking: If a passage consists of three completely independent facts, the prior context should not "help" the model and thus not lead to a lower loss at later positions. As this is a crucial building block for the entire paper, this issue should be assessed directly.
-
The concerns of point 1 get even stronger when looking at the results for Figure 2. Here, FOVEATED leads to fairly consistent improvements across all questions. This seems counterintuitive given the underlying motivation, as this would suggest that FOVEATED mainly provides improvements for later questions (i.e., questions about facts present in the later parts of the paragraph). Additionally, it is unclear to me how FOVEATED can provide substantial improvements to the first question. Given that there should be only limited context prior to the related fact, FOVEATED should intuitively have much less impact here.
-
While I appreciate the analysis of general capability preservation, the evaluated benchmarks do not provide a holistic assessment of the "general capabilities" of a modern LLM. The authors should either conduct a more holistic assessment (e.g., using an evaluation suite like CapTrack [1]) or re-scope this section.
-
The case studies would be much stronger if the authors provided the corresponding passage as well. This would allow better insights into how the proposed method qualitatively improves the model's responses.
-
In Figure 2, it is very difficult to accurately assess the actual difference between the solid and transparent bars. The authors should explore alternative visualizations (e.g., a bar for the base performance and a marker for the proposed method) to make the figure easier to read.
[1] Thede et al., 2026, CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Questions for authors
- Do the authors see the issue of context-induced difficulty underestimation specific to unstructured knowledge editing, or is this a general issue in language model pretraining (i.e., that factual content occurring later in a given block is trained on less)?
6 comments
Corresponding passages in the case studies.
As suggested, we added the editing request and the complete passage for the case studies in the main text in the revised paper.
Improving readability of Fig. 2.
We thank the reviewer for their suggestion. We have updated the figure (now Fig. 3) accordingly.
Context-induced difficulty underestimation beyond unstructured knowledge editing.
Thank you for bringing up this interesting question. Conceptually, we agree with the reviewer and hypothesize that this issue is not specific to UKE: it may arise in more general language modeling whenever the content within a text is related, so that the preceding content makes the later content easier to predict. UKE offers a clear setting to study the issue: the edited knowledge is usually counterfactual and unknown to the model by construction. This helps rule out the possibility that the model already knows part of the new knowledge. This possibility is harder to exclude in more general settings, which makes the study more challenging but also worth pursuing. We have revised our draft to include this discussion.
Thank you for the comprehensive and clear rebuttal. The clarifications and new analyses address most of my concerns. I have one follow-up about the role of semantic relatedness in the proposed mechanism.
I understand that passages in UKE benchmarks often describe multiple aspects of the same subject. Even within such passages, however, some facts may provide little information about one another. The sentence-shuffling analysis shows that the same sentence tends to receive lower pre-edit loss when placed later in its original passage. How much of that reduction depends on the preceding sentences being semantically related to the target fact?
One way to test this would be to replace the original preceding sentences with a length-matched prefix from an unrelated passage, while holding the target sentence, editing prompt, and target token positions fixed. Comparing pre-edit loss under the two prefixes would help isolate the contribution of related context. It would also be informative to compare FOVEATED’s gain under both conditions. Does the gain diminish when the unrelated prefix provides less support for predicting the target fact, or does FOVEATED yield a similar improvement?
We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one, which will also be updated in the revised paper.
Role of the preceding context in the lower loss at later positions.
Thanks for your question. We would like to clarify that in UKE benchmarks (including the ones we use), the free-form passages often describe a subject and multiple aspects of this subject, each of which can be seen as an atomic fact [1, 2, 3]. These facts are not independent facts from different domains. (Following the reviewer's later suggestions, we included some passage examples in the revision.) Therefore, in such passages, the preceding content is related to the later facts, and context-induced difficulty underestimation thereby arises.
In App D.1 we showed that the later position tokens, regardless of how different sentences are shuffled, tend to have lower losses. Following the reviewer's thoughts, we further compared the loss of the same sentence when it is placed at different positions. As shown below, the same sentence receives lower loss when it is placed later, i.e., with more preceding context. This indicates that in general, a sentence can benefit from more of its preceding context (from the given passage, not an arbitrary one) and have a lower initial loss if it is placed in a later position of the passage. We have added these results to the revised paper.
Tab. Statistics of sentences when placed at later positions (more details are in App. D.1 of the revised paper)
[1] Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models, 2025
[2] Uncovering Context Reliance in Unstructured Knowledge Editing, 2026
[3] FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing, 2026
We thank the reviewer for the careful reading. We would like to clarify that the first fact is also learned with context given by the editing prompt that introduces the subject. Additionally, this editing prompt is often different from the first atomic question. As a result, when FOVEATED perturbs the view of the prompt in the same way as the preceding sentences, the first fact benefits from the context-shortcut-breaking mechanism as well.
In practice, we view the position of a fact as a practical proxy for how much it may suffer from context reliance. To better quantify how atomic facts may benefit from FOVEATED, we compared facts at the same position with low and high contextual support, and found that the latter usually benefit more from FOVEATED (e.g., +10.0 vs. +5.7 on UnFine-CF with Qwen2.5). We have added these results to the revised paper.
We thank the reviewer for pointing us to CapTrack [1]. In this work we follow the convention in UKE literature [2] to evaluate the edited models' general capabilities, and found that the shortcut-breaking objective induced by FOVEATED brings, on average, modest influences on top of the editors' impacts. Following the reviewer's suggestions, we conducted additional experiments (under the same setting as our ablation studies) on a subset of CapTrack tasks spanning the three categories (CAN, WILL, and HOW). Results are shown in the table below. We found that FOVEATED brings both modest improvements and degradations on these tasks, similar to our observation on the six benchmarks. We have revised our paper and included these results.
[1] CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training, 2026
[2] FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing, 2026