acceptodds
Under review as a conference paper at ICLR 2027

How Do Language Models Choose Between Context and Memory?

Abstract

When contextual information conflicts with knowledge stored in model parameters, activation directions can be used to decode and steer which source the model follows. However, successful steering does not establish that the unedited model uses those directions to choose between sources, or that they remain effective across tasks. To test these possibilities, we vary the stated authority of contextual claims while holding their content fixed. We first estimate authority directions from prompts in which context and parametric knowledge agree, then test their causal contribution when the two sources conflict. Interchanging naturally occurring activation values along these directions between matched high- and low-authority prompts reproduces 30-68% of the authority-induced shift in source choice across Qwen, Llama, and OLMo models, whereas matched controls reproduce almost none. We next ask what transfers across tasks: the learned direction versus the activation values exchanged along it. Using a direction learned on another task closed 9% of the source-choice gap, compared with 57% when learned on the task being evaluated. Both interventions exchanged activation values from the evaluated task. In a separate experiment, we kept its learned direction but exchanged values taken from another task, which closed 68% of the gap. These results show that authority-related activation values can causally influence source choice across tasks when inserted along directions learned for the task being evaluated.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2KeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how language models choose between contextual information and parametric memory when the two sources conflict. The authors extract authority-related directions using prompts with different authority cues and evaluate their causal effects through activation steering and activation interchange. Experiments on Qwen, Llama, and OLMo suggest that these directions can influence source selection within individual tasks, while their cross-task transferability remains limited.

Critical strengths

The paper makes a useful distinction between representational decodability and causal effectiveness, rather than inferring model mechanisms solely from linear probing or steering results. The evaluation also covers multiple model families and includes random and covariance-matched controls, which strengthens the empirical analysis.

Critical weaknesses

My main concern is that the paper does not sufficiently explain how the model actually chooses between context and memory. Showing that certain activation directions can influence the final prediction does not necessarily reveal the underlying selection mechanism. In particular, the paper provides limited insight into how these directions emerge, which computational components produce the relevant signals, and how these signals contribute to the final decision. The current contribution appears closer to a causal evaluation of activation steering than a mechanistic explanation of source selection.

Figure 3 raises an important concern. Increasing the intervention dimensionality substantially improves mediation, but randomized and shuffled-label controls achieve comparable or even stronger effects. This suggests that improved intervention performance may not originate from authority-specific information, but instead from the increased dimensionality of the manipulated representation. Although the authors acknowledge this limitation, it weakens the interpretation of the results as evidence for a specific authority mechanism.

The experiments primarily focus on colors, object materials, and state/country facts, with many evaluations relying on next-token prediction over fixed candidate sets. While these settings provide experimental control, they are considerably simpler than context-memory conflicts encountered in realistic language model applications. Real-world scenarios often involve longer contexts, multiple conflicting sources, and more complex reasoning.

Questions for authors

see Weaknesses

Reviewer_s4fnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

When contextual information conflicts with knowledge stored in model parameters, activation directions can be used to decode and steer which source the model follows. However, successful steering does not establish that the unedited model uses those directions to choose between sources, or that they remain effective across tasks. The paper estimate authority directions from prompts in which context and parametric knowledge agree, then test their causal contribution when the two sources conflict. The paper also explores what transfers across tasks: the learned direction versus the activation values exchanged along it. The paper shows that authority-related activation values can causally influence source choice across tasks when inserted along directions learned for the task being evaluated.

Critical strengths

(1) The paper is related to an important real-world use case and motivated clearly. The paper explores a fundamental area related to this topic.

(2) The paper has very detailed analysis results. Some of the results are very insightful.

Critical weaknesses

It might be difficult to read the paper and reproduce some of the results for the readers who follow this work.

Some keywords are mentioned but not very clear according to the descriptions. For example, related to "We also experiment over five authority cue families: authority wording, source credibility, cited evidence, consensus, and first-person confidence as an empirical null baseline.", it is not clear how they are defined and reliably calculated or measured in practice. There is limited information about them in the paper. The choices of the five authority cue families seem to be arbitrary. It would be great to cite some papers or provide the rationale of these choices.

It is not very clear which datasets are evaluated in the experiments. It would be great to use some public benchmark datasets. It seems that all the baselines are from this paper and not from any previous papers. It would be great to have some discussions related to the choices of the datasets and baselines. It might be helpful to have a subsection dedicated for all the details and setup information related to the experiment.

Section 4 might be very brief and provide limited information.

In addition to the quantitive evaluation and analysis, it might be helpful to add some case study results.

Questions for authors

It would be great to provide some details.

(1) Related to "We also experiment over five authority cue families: authority wording, source credibility, cited evidence, consensus, and first-person confidence as an empirical null baseline.", it is not clear how they are defined and reliably calculated or measured in practice. There is limited information about them in the paper. The choices of the five authority cue families seem to be arbitrary. It would be great to cite some papers or provide the rationale of these choices.

(2) Is there any public dataset used in the experiment? If not, would it be possible to evaluate the developed approach on public datasets?

(3) Are there any baselines from previous papers, so the positioning of this paper is more clear?

(4) Would it be feasible to present some case study results, in addition to the quantitive evaluation and analysis?

Reviewer_1FywOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how instruction-tuned language models resolve conflicts between parametric memory and contextual claims, focusing on “authority” as a source cue. The authors estimate linear authority directions from agreement prompts, apply additive steering and coordinate interchange on conflict prompts, and test whether these directions and activation values transfer across tasks and authority cue families. Experiments are conducted on Qwen2.5-3B/7B, Llama-3.1-8B, and OLMo-2-7B using synthetic color, material, and state/country tasks. The main claims are: (1) authority-related activations causally influence source choice within a task; (2) activation values can transfer across tasks when inserted along a receiver-task direction; (3) learned directions transfer poorly across tasks; and (4) high cross-task decodability does not imply causal use.

Critical strengths
  1. The paper separates additive steering, natural-coordinate interchange, and cross-task transfer. This is a useful distinction, and the negative result that decodable authority information need not be causally used is valuable.

  2. The authors include random directions, covariance-matched controls, authority-breaking controls, and held-out subject evaluations. The mediation and gap-closure analyses are more careful than a simple probing study.

  3. The finding that color-task activation values can influence material-task source choice when inserted along a material-trained direction, while direct reuse of the color direction is weak, is conceptually useful.

Critical weaknesses
  1. The experiments only cover Qwen2.5-3B/7B, Llama-3.1-8B, and OLMo-2-7B. These are relatively old/small models by current standards, and it is not clear whether the main conclusions hold for newer or larger instruction-tuned models. At minimum, the paper should justify this choice and discuss the risk that the findings are model-specific.

  2. Mean-difference directions, additive steering, and coordinate interchange are well-established tools. The paper applies them carefully, but I do not see a substantial methodological or theoretical advance. The contribution feels more empirical than methodological, which is a concern for ICLR.

  3. Random, covariance-matched, and authority-breaking directions are useful, but there is little comparison with stronger alternatives such as DAS-style causal abstraction, SAE feature interventions, causal tracing, or representation fine-tuning. Without such baselines, it is hard to know whether the estimated authority direction is the right causal object.

  4. Small answer sets and templated prompts make the experiments clean, but they limit external validity. It is unclear how well the results transfer to naturalistic knowledge conflicts, longer contexts, or generation tasks.

Questions for authors

See weaknesseds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.