- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies if large language models think based on the original source of intermediate values or restated copies, which are generated in the middle of chain-of-thought reasoning. It is demonstrated that having correct local copies reduces the sensitivities of the models (e.g. Gemma-2, Qwen3). In a controllable experimental setup, it is shown that this preference is caused by the reliability and frequency of the copies in training.
Critical strengths
- The paper proposes a clear method to modify the internal memory to investigate the effect of the intermediate copy (e.g. replacing X=10 by X=50 to see how the reasoning outcome will differ). The idea of having intervention in the memory mechanism is interesting. Extensive ideas could be developed on this idea, e.g., the safety and adversarial attacks (how to make modes robust against such memory modification and injection).
Critical weaknesses
- The writing and presentation quality is, unfortunately, suboptimal and the paper requires a major revision. A paper should primarily present a clear research question, a methodology how to solve it, and insights obtained through the experimentation (or theoretical formulation). In the current form of the paper, it is challenging for readers, may not impossible, to extract those essence from the manuscript, as many parts of the explanation need assumptions, which are not introduced in the paper in a self-contained manner, to interpret.
- The evaluations is conducted only on synthetic test environments e.g. with arithmetic templates. Thus a major concern remains on the applicability of the study. Although I appreciate the idea, the evaluation is not supporting the introduced claim sufficiently.
Questions for authors
- Although the paper touches upon scenarios such as basic words, names, and PrOntoQA deductions, how can the findings be applied to the general, unstructured reasoning?
2 comments
Authors Response to Reviewer VDwF (Part 2 of 2)
Q. How do the findings apply to general, unstructured reasoning?
To answer your question, we tested this directly with free-form two-hop answers over Wikipedia passages (MuSiQue), in which no arithmetic and no template is involved (Sec. 6, final paragraph; Appendix A7.5, Table A45). We keep the sentence in which the model names the intermediate entity, remove the passage that states it, and append one bridging sentence that either names the entity again (a copy) or refers to "the one just identified" (a reference). The model then continues in whole sentences.
| Model | Reference: source replaced, next inference changes | Correct copy: source replaced, next inference changes |
|---|---|---|
| OLMo-2-7B | 42.2% | 4.5% |
| Qwen3-8B | 60.0% | 3.2% |
| Gemma-2-27B | 57.9% | 12.7% |
A correct copy shields the next inference from a change of the source in all three models, as in arithmetic. We report the limits of this result in the manuscript.
- Final answers return to the entity of the source sentence more often than the next inference does.
- Gemma-2-2B reaches neither answer in more than half of its continuations and is not interpreted.
- When the intermediate entity is also stated in the supplied passages, edits to its mentions in the trace do not carry beyond their own sentence (Appendix A7.6).
We therefore claim the following for unstructured reasoning: a restated intermediate value shields the next step from its source, and a step that refers back without restating remains sensitive to it. We do not claim that a changed copy determines the final answer of a free-form response.
On robustness to modified or injected intermediate values
You note that the intervention suggests further work on robustness to memory modification and injection. The fine-tuning result in Sec. 7 bears on this point. By default, a value altered at its restatement decides the next calculation on 95 to 98% of dependencies in the models' own GSM8K solutions. After fine-tuning on solutions in which restated values are sometimes wrong, the altered restatement decides at most 1%, and the models compute from the source on 86 to 99%.
Changes made in response to your review
- Abstract and Introduction: research question stated explicitly; findings summarized in section order; section pointers in the contributions.
- Figure 1: concrete source and consumer lines with the measurement stated in one line.
- New Sec. 6: restated values in model-written GSM8K solutions, prompt variants, and free-form two-hop answers.
- Sec. 7: reliability fine-tuning of three pretrained models on GSM8K solutions.
- Limitations: the boundaries listed above.
Code and result files for all new experiments are in the anonymous repository.
Authors Response to Reviewer VDwF (Part 1 of 2)
Thank you for the assessment and for your interest in the intervention itself. You raise two concerns: that the research question, method, and insights are hard to extract, and that the evaluation is limited to synthetic environments. The revised manuscript addresses both; new text is set in blue. We respond to each point in the order of your review.
W1. The research question, methodology, and insights are hard to extract
We agree that the submission did not state these three elements plainly. The revision states each in one place.
x3 = 14 + 9 = 23) and consumer line (x5 = x3 - x1 = 23 - 16 = 7) with the source, the local copy, and the prediction marked, and states the measurement in one line. Sec. 3 defines the source intervention, the comparison with and without a copy, and what the comparison does and does not measure.Self-contained definitions. The three terms on which the paper relies (source, consumer, local copy) are defined in the first paragraph of the Introduction on the example line shown in Figure 1. Terms needed only for the method are defined at first use in Sec. 3. The paragraph "What the comparison measures" in Sec. 3 states the assumptions under which a small source effect should be read: it is conditional on the supplied text and does not show that the source is absent from the model's internal state.
W2. The evaluation uses only synthetic test environments
You are right that the submission evaluated only assigned formats and templates. The revision adds a section on reasoning that the models write themselves (Sec. 6) and extends the training experiment to pretrained models and natural solutions (Sec. 7).