acceptodds
Under review as a conference paper at ICLR 2027

MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

Abstract

Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad , where familiar content improves reasoning performance, and the , where models skip reasoning entirely and recall stored answers. To test these effects, we introduce , a human-curated benchmark that pairs factual reasoning tasks with structurally identical fictitious versions where real entities like people, companies, or dates are systematically replaced by fictitious ones of the same type. This preserves task structure and specified reasoning operations while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7% in the fictitious setting, demonstrating a clear memory bias. However, a targeted analysis of questions failed in the fictitious setting shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious evaluation to broader reasoning settings.

Primary area:
datasets and benchmarks
Keywords:
Parametric memory, contextual reasoning, evaluation, knowledge conflicts, reasoning benchmarks, reasoning robustness, reasoning shortcuts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nmc2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemoReason, a paired question-answering benchmark constructed by replacing entities in factual Wikipedia-based texts with fictitious entities governed by consistency rules, so that task structure is preserved while familiarity with the content varies. The authors design variant-answer, invariant-answer, and refusal questions, together with extractive, arithmetic, temporal, and inference tasks, to compare model performance on familiar factual texts and their fictitious counterparts. Experiments show that reasoning performance generally declines on fictitious texts, but errors rarely directly reproduce the corresponding factual answers, leading the authors to argue that a strong shortcut hypothesis based on skipping reasoning and recalling memorized answers is not the dominant failure mode. The paper further examines the effects of partial entity replacement, chain-of-thought, and reasoning effort on the performance gap, and releases the dataset, code, and annotation workflow.

Critical strengths
  1. The paired factual/fictitious design provides clear experimental control, helping to distinguish changes in content familiarity from changes in task structure.
  2. The benchmark construction incorporates human review, entity-type annotation, consistency-rule constraints, and public resources, making the work relatively transparent and reusable.
  3. The paper goes beyond reporting an overall gap by analyzing it across question types, replacement proportions, reasoning settings, and direct answer reproduction.
Critical weaknesses
  1. Although fictitious entities are type-matched, they may still introduce hidden changes in linguistic distribution, readability, plausibility, or question difficulty, and these confounding factors are not sufficiently ruled out.
  2. The interpretation of “memorization bias” remains mainly behavioral, without more direct mechanistic evidence, which makes the causal conclusions difficult to support strongly.
  3. The tasks are focused on Wikipedia-style factual texts and a limited set of question types and models, so the generalization to broader contextual reasoning capabilities remains insufficient.
Questions for authors
  1. How do you rule out the possibility that the performance change is driven by differences in the linguistic distribution, readability, or plausibility of the fictitious entities themselves, rather than by familiarity with parametric memory?
  2. Beyond observing final output behavior, have you considered using internal representations or stronger interventional experiments to verify the causal mechanism behind the proposed memorization bias?
  3. Since the benchmark mainly relies on Wikipedia-style factual texts and a limited range of question types and models, on what basis do you believe the conclusions remain valid in more open-ended contextual reasoning settings?
4 comments
AuthorsOpenReview ↗

Additional Experiments on Readability, Task Difficulty, and Familiarity, and Clarification of our Conclusions

We thank you for your constructive assessment and for recognizing the design's "clear experimental control," the benchmark as "relatively transparent and reusable," and the analysis that "goes beyond reporting an overall gap." We address the three concerns below.

W1 & Q1: "[...] differences in the linguistic distribution, readability, or plausibility of the fictitious entities themselves, rather than by familiarity with parametric memory?"

Thank you for identifying these possible confounds. Our construction limits these effects through type-compatible substitutions, consistency rules, coherent entity attributes, and double blind human validation. To further investigate these issues, your feedback motivated the additional checks below.

1. Readability and plausibility:

Our generation prompt explicitly requests readable and plausible entities (l.1705–1706):

[...]
- they should read like plausible names while still looking clearly invented.
- Follow ordinary English phonotactics. Avoid impossible consonant clusters, unreadable punctuation, excessive doubled letters, and fantasy-style spellings.
[...]

Following your suggestion, Flesch-Kincaid grades are 14.25 (factual) and 14.47 (fictitious): +0.22 [95% CI: −0.014, 0.43], a small, nonsignificant difference. We will add this analysis of human readability to the camera-ready version.

2. Task difficulty:

Replacements are sampled under template rules that constrain value ranges and preserve order, offsets, and arithmetic relations, so instantiation difficulty varies symmetrically around the factual values rather than in a principled direction, and every reported number averages ten variants. To address this concern (also raised by Reviewer 9dwv), MemoReason includes a configuration where only named entities are replaced and all numerical/temporal entities are kept unchanged, leaving the numerical difficulty unchanged. Table 8 (Appendix G.1.2, in blue in the revision) shows significant drops for five of eight models: therefore, harder numerical calculations cannot be the only explanation.

3. Linguistic distribution:

We agree with the reviewer that there is a shift in the linguistic distribution, and this is exactly what MemoReason intends to test: what happens to LLMs’ performance when familiar content is replaced with unfamiliar content while the task structure is preserved? Our hypothesis is that reduced familiarity may contribute to the observed performance drops.

Using Qwen3.5-27B, we computed mean document log-likelihood from the probabilities assigned to successive tokens given their preceding text. These probabilities reflect patterns learned during training, making log-likelihood a model-relative indicator of statistical familiarity. We also measured the mean token length of the annotated entity surface forms:

MeasureFactual, mean ± SDFictitious, mean ± SDFictitious − factual [95% CI]
Mean log-likelihood (nats/token)−1.424 ± 0.372−2.547 ± 0.332−1.124 [−1.198, −1.048]
Mean entity length (tokens)3.256 ± 0.7684.252 ± 0.907+0.997 [+0.902, +1.095]

Fictitious documents have significantly lower log-likelihood, consistent with reduced statistical familiarity under this model, and their entities are longer on average. These diagnostics quantify the distribution shift without establishing the cause of the performance gap. However, changes in log-likelihood show a weak, nonsignificant correlation with performance drops across documents. Larger measured shifts do not consistently correspond to larger performance losses.

W2 & Q2: "[...] using internal representations or stronger interventional experiments to verify the causal mechanism [...]"

We agree that our evidence is behavioral rather than mechanistic, consistent with the scope of this datasets and benchmarks paper. As noted in the conclusion (line 477): "Identifying the mechanisms underlying these performance differences remains an open question [...]". To support this research, we release our annotation interface, code, and all MemoReason configurations (109,200 examples; anonymous links in page 1’s footnote).

W3 & Q3: "[...] on what basis do you believe the conclusions remain valid in more open-ended contextual reasoning settings?"

Thank you for raising this point. We agree that our results on Wikipedia-based QA do not establish generalization to open-ended reasoning, and we will explicitly mention this limitation in the paper. Our released annotation interface and guidelines make it easier to extend MemoReason to other text genres and tasks, and to test whether our findings hold in those settings.


Note: we have also conducted new experiments on chain of thought effect (see answer to reviewer 9dwv).


Thank you again for helping us strengthen the evaluation and clarify our conclusions. We hope these clarifications and new experiments will support a higher assessment.

AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer nmc2,

As the discussion period draws to a close, we would like to kindly follow up on our response of 11 November.

We really appreciate your thoughtful feedback, which helped us strengthen the evaluation and clarify our conclusions. Following your suggestions, we provided additional experiments of readability and linguistic differences between the factual and fictitious texts, as well as a control experiment for task difficulty (which we added to the paper -- Table 8 in blue). We also clarified what our results do and do not establish about the causes of the performance gap and whether the findings extend to other reasoning tasks.

Could you please let us know whether these additions and clarifications address your main concerns, or whether any points remain unresolved? We would be happy to provide any further clarification.

Thank you again for your time and consideration.

Thank you for the author's serious response. The author's response has largely resolved my concerns about reliability and generalization, so I have raised my score to 3

AuthorsOpenReview ↗

Big Thanks! :)

We sincerely thank Reviewer nmc2 for the positive follow-up and for recognizing that our additional experiments and clarifications have largely addressed the concerns about reliability and generalization. We greatly appreciate the increased score and the reviewer's thoughtful engagement with our rebuttal. Your constructive feedback has helped us strengthen the evaluation and clarify the scope of our conclusions, and we will ensure that these improvements are clearly reflected in the revised manuscript.

Thank you again for your time and valuable feedback!

Reviewer_1rCnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how we can separate out the effect of parametric memory (specifically memorized facts) from reasoning or quetion answering. The paper introduces a new dataset of replacing wikipedia entities with other entities and evaluating LLMs on counterfactual questions which would/would not be changed by changing those entities. They find that modern LLMs still show a gap in performance on replaced data.

Critical strengths

The overall idea of separating out complex reasoning tasks from the parametric knowledge of LLMs is a generally useful idea (but this is not unique to this paper see below).

The experiments themselves seem mostly pretty thoughtful and correct as far as I can tell. Absent prior work (discussed below) I would say these experiments are well designed to find interesting insights about how parametric knowledge affects LLM performance on reasoning-related QA tasks.

Critical weaknesses

The biggest weakness of this paper, is that both the idea for this kind of dataset, the phenomena of memorizing or anchoring to known facts, and even very specifically doing both of those things on Wikipedia are extremely well known in the literature and this paper barely mentions prior work and does not even have a prior work section. It briefly mentions but does not differentiate from several really important prior works (especially Longpre et al. (2021)) and on the papers it does briefly mention, it does not do an adequate job of differentiating what this paper does differently.

First, the phenomena itself is really well known. There is actually a survey paper on this exact topic: Xu et al., 2024. Knowledge Conflicts for LLMs: A Survey. EMNLP 2024. This huge line of work is barely engaged in at all other than two very short paragraphs in the intro.

The dataset itself is also not all that unique. Again, I think the idea of the dataset and the dataset itself is extremely similar to NQ-Swap. It takes existing wikipedia passages, does entity replacement and measures language model's changing capabilities on these new questions. The paper does not directly differentiate from it, and as far as I can find, the only acknowledgment and differentiator from prior works is the line that "the gap they reported could be fully explained by... the fact they used two distinct sets of evidence-question pairs" there referring to CofCa. They do not differentiate from any of the other datasets including NQ-Swap, and even this differentiator they do have seems very weak to me. Again, this paper should have a full related works section that does all of this.

On the dataset itself, despite the title and the claim in intro, these questions do not really seem to require very much reasoning at all and are not complex. In fact, NQ, which was released in 2019 seems about the same level of difficulty as this one, and that dataset was also written by humans rather than partially AI-generated, so it feels like we are actually going backward in terms of dataset quality. And unlike other datasets like HotPotQA, it doesn't require more than an extra hop of reasoning. The questions just do not seems different enough to differentiate it from the many many many QA datasets on NQ, so again, going back to NQ-Swap, I do not really see this dataset as providing much compared to that.

On the dataset construction, it seems that the method is to do all of these entity replacements with an LLM to produce everything. And as authors point out in human validation, this required a lot of cleanup by humans. Given this, I do not know why none of the very long prior work on entity linking and replacement, the authors did not consider or use any of this. It feels like authors just reached for LLMs as the tool and did not consider other, maybe more effective techniques.

On the experiments, I think a lot of these made a lot of sense, but again, the conclusions here are extremely well known in the LLM community. Changing familiar entities to unknown, false ones, on Wikipedia, which LLMs usually oversample in pre-training, hurts their performance on knowledge-based questions. And, in fact since those earlier works, it seems like QA has in general improved a ton because the actually topline numbers (claude gets 92% on the fictitions set) so even now, the dataset seems to really be getting saturated.

Questions for authors

None

6 comments
AuthorsOpenReview ↗

Prior Work Focuses on Knowledge Conflicts and we Focus on Familiarity (this distinction is crucial)

We thank the reviewer for acknowledging that the paper is "clear and easy to follow" and that "the experiments themselves seem mostly pretty thoughtful and correct.". Your review states: "Absent prior work (discussed below) I would say these experiments are well designed to find interesting insights." The assessment therefore rests on a single factual question: does prior work, NQ-Swap in particular, already do what MemoReason does? We answer it concretely below.


"[...] the phenomena itself is really well known. There is actually a survey paper on this exact topic: Xu et al., 2024. Knowledge Conflicts for LLMs: A Survey. EMNLP 2024. [...] the dataset itself is extremely similar to NQ-Swap [...]"

We respectfully disagree as these two benchmarks perform different interventions and measure different quantities:

  • NQ-Swap (Longpre et al., 2021) substitutes the gold factual answer entity in a passage with a different real entity, deliberately creating a conflict with a memorized fact, and measures whether a single-hop extractive model answers from context or from memory.
  • MemoReason (ours) replaces all entities with replacements verified to be non-existent (web and Wikipedia checks, Section 2.3), under template rules that preserve the reasoning operations, and pairs every document with its factual twin. Because the fictitious entities have no counterpart in training data, there is no stored fact to conflict with and therefore MemoReason isolates the effect of familiarity on reasoning in the absence of conflict. The conflict literature, including the survey you cite (Xu et al., 2024), does not measure this quantity.

The clearest evidence that these are different objects is that MemoReason contains the NQ-Swap style question as a diagnostic. The Shortcut Rate of Section 4.4 asks exactly whether failed fictitious answers reproduce the memorized factual answer (the mechanism NQ-Swap is built around) and finds rates of only 1.0–3.2% (Table 4). The phenomenon NQ-Swap studies is thus tested within our benchmark and excluded as the explanation of the measured drops.

Knowledge conflict itself appears in MemoReason exactly where we introduce it deliberately: the partial-replacement regime of Section 4.2, explicitly anchored to Longpre et al. (l.382–384), where the conflict regime scores below full fictionalization (Figure 2) showing that the two effects are empirically distinct in our own data. We will cite Xu et al. (2024) and position the paper against the survey along precisely this line.


"[...] this paper [...] does not even have a prior work section [...] the only acknowledgment and differentiator from prior works is the line [on CofCa] [...]"

The paper already contains a dedicated Related Work section (Appendix A, l.681–733), which discusses the counterfactual and conflict literature and contains the task-parity differentiation of prior counterfactual benchmarks (l.705-717). Its placement reflects the main text page limit, not an omission. Together with the introduction, it discusses a substantial body of relevant literature, including Longpre et al. (2021), Wu et al. (2024a), Wu et al. (2024b), and Mirzadeh et al. (2025). Following your comment, we nevertheless agree that the distinction with NQ-Swap deserves greater prominence and have added a paragraph (in blue, l.718–733) distinguishing knowledge conflict from unfamiliarity without conflict, and will add a short related-work paragraph to the main text using the extra camera-ready page.


"[...] the conclusions here are extremely well known in the LLM community [...]"

What is well known is model behavior under knowledge conflict. To our knowledge, besides the works cited in the paper (i.e. GSM-Symbolic by Mirzadeh et al. 2025, MATH() by Srivastava et al. 2024, and CofCA by Wu et al. 2024a), few studies isolate the familiarity effect without introducing knowledge conflict or changing the task structure, and none provides a factual twin for every reasoning item.


If there is an existing dataset that pairs each reasoning task with a factual twin in the absence of knowledge conflict, beyond those cited, we would welcome the pointer and will gladly cite and compare against it.


References:

  • Longpre et al. 2021. Entity-Based Knowledge Conflicts in Question Answering. EMNLP 2021
  • Xu et al. 2024. Knowledge Conflicts for LLMs: A Survey. EMNLP 2024
  • Wu et al., 2024a. CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark. ICLR 2025
  • Mirzadeh et al. 2025. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. ICLR 2025
  • Srivastava et al. 2024. Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap. Preprint
  • Wu et al., 2024b. Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. NAACL 2024
AuthorsOpenReview ↗

Task Complexity, Dataset Construction, and Benchmark Saturation

In this second comment, we address your remaining concerns about task complexity, dataset construction, and benchmark saturation.

"[...] these questions do not really seem to require very much reasoning at all [...] NQ, which was released in 2019 seems about the same level of difficulty [...]"

Our objective with MemoReason is not to maximize reasoning difficulty, but to identify significant drops due to familiarity effects while preserving the reasoning task and document structure. Even so, MemoReason goes beyond the answer extraction setting used in NQ-Swap. Extraction serves as a separate control class in MemoReason (see Table 2's Extr. column) precisely because it involves no composition, while the arithmetic, temporal, and inference questions require combining several pieces of information under the document's rules. The short examples in Figure 1 illustrate the replacement procedure, not the full range of reasoning required. The benchmark remains challenging for recent models (fictitious accuracy of 66.42% for OLMo-3-7B-Instruct, 76.88% for Llama-3.1-8B-Instruct, 87.65% for GPT-OSS-20B -- see Table 5).


"[...] I do not know why none of the very long prior work on entity linking and replacement, the authors did not consider or use any of this [...]"

Classical entity linking resolves mentions to existing entities, whereas our construction requires generating non-existent, pronounceable, type-compatible replacements whose linked attributes (e.g., a place name and its demonym) remain coherent across mentions. It also requires document-specific rules governing arithmetic relations, offsets, and value couplings (Section 2.2 and Appendix B.2), which no entity-linking toolkit produces. The LLM supplies pre-annotations only: 15 human annotators reviewed every item in two independent passes, with an initial inter-annotator agreement of 76.22% and all disagreements resolved in consensus sessions. Roughly 50% of Q-A pairs required human intervention, every variant passes deterministic rule validation, and the process took about 200 hours (Section 2.2). The dataset is human-curated with AI assistance, and we report the correction rate transparently as part of its quality record.


*"[...] claude gets 92% on the fictitions set [...] the dataset seems to really be getting saturated."

High accuracy does not imply saturation for a robustness benchmark. As Section 4.5 states (l.447-452), high accuracy alone does not establish it. Claude Sonnet 4.6 reaches 92.30% accuracy on fictitious items compared with 95.67% on factual items. This gap is statistically significant under document-clustered 95% confidence intervals and corresponds to an error ratio of 1.78$\times$, meaning that removing familiarity nearly doubles its errors. The evaluated pool spans 66.4-92.3%, the regime where the benchmark discriminates most, and the templates regenerate fresh fictitious instances, mitigating leakage-driven saturation.


In conclusion, we hope that these clarifications have strengthened the novelty of our contribution and will support a higher assessment.

AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer 1rCn,

As the discussion period draws to a close, we would like to kindly follow up on our responses of 11 November.

We really appreciate your feedback, which helped us clarify the contribution of our work and its relationship to prior research. In our responses, we explained how MemoReason differs from NQ-Swap and added an explicit comparison in the related-work section. We also clarified the reasoning required by our tasks, the role of human annotators in building and validating the dataset, and why the benchmark remains informative even for models with high overall accuracy.

Could you please let us know whether these clarifications address your main concerns, or whether any points remain unresolved? We would be happy to provide any further clarification.

Thank you again for your time and consideration.

Reviewer_9dwvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

In this paper, they want to see does parametric memory has an effect on the contextual reasoning of LLMs and how it does, and also do models rely on memorized info or rely on reasoning on the provided context. They do this with introducing a benchmark and named it Memoreason, and it uses factual documents from Wikipedia and then generates fictitious versions by replacing entities (keep the document structure and reasoning operations). Their benchmark has 12k question-answer pairs that come from 100 document templates with 12 questions, and for each of them they generate 10 fictitious versions. Their questions have four different types and three different answer types. Their results show that performance is lower when entities are fictitious compared to when entities are factual (mostly on reasoning type questions and variant type answers). They also do this with different percentages of replacing the entities, CoT, and reasoning effort, and see that do models recall factual answers when they fail in a fictitious setting.

Critical strengths

Their design is solid because they are considering to use paired factual and fictitious documents instead of just comparing two different datasets, and it is novel, as they mentioned in 1, previous works they discuss uses differences between factual and fictitious, and it is a bottleneck because these two do not have the same reasoning structure, and the difference in their performance can come from different task difficulties, and it should not be considered in one group. So the way that they address this in 2, with the same template and replacing entities under different rules to keep the logical relationship the same, is solid. Also, the way that they consider four different question types and three different answer types is solid because it helps to see do different reasoning tasks are affected by familiarity and how. In 4.2, the idea that they also consider different percentages of replaced entities is also solid. Also, the way that they consider the difference between memorization bias and the Strong Parametric Shortcut Hypothesis is solid because they do not conclude that every performance drop comes from that the model is recalling its answer that has been memorized. Also, in 4.5 the way that they use different metrics is solid.

Critical weaknesses

It is not convincing that the performance drop is just because of parametric memory. In 2 they keep the same structure and operations, but then when they replace the entities, it also changes the numbers, names, dates, and also sometimes the correct answer. So it is not clear that do the models perform lower because that the info is not familiar to them, or because that the entities that have been replaced make the questions more difficult. For example, in figure 1, the factual requires 1968-1960, but the fictitious requires 1981-1950. So even that both have the same operation, it does not mean that they have the same difficulty for the model. In 4.1 and table 2, extractive questions with variant answers show performance drops for all models, but since this question type needs finding information besides its reasoning, so it is not clear that the drop is because of the reasoning or that the information is not familiar.

The thing that they mention about the reason for recovery in figure 2 is not convincing, they explain it by saying that it is because of the knowledge conflict between the factual information that is still in the document and the fictitious information. Because for example, when they do the replacing of entity randomly, it does not mean that the same important factual information is kept. For example, when you keep the name of a person but you change the date that can make a different case. So the result can be dependent on that what entities are replacing.

In 4.3, they compare olmo-3-7b-instrcut with olmo-3-7b-think but these two are different post-trained checkpoints, so the performance difference can not just be that it comes from CoT.

On the side note in line 303, it is good to refer that it is the delta in line 271.

Questions for authors

In line 333, the range that you are saying is too much and might sound like sometimes 4, sometimes 15. Can you report the median instead?

In the exact match and judge (evaluation framework), why are you not passing everything to the judge, because in this way, how can you be sure that the extraction is correct and that something else has not been extracted that is used for exact match? I mean the extraction, extract a correct thing. Having a uniform evaluation is better, or maybe just make sure all of the things that have been extracted are correct.

In 2.2, can you report the inter-annotator agreement?

In 2 and 4.1, can you give a more direct control to show that the performance drop is because of parametric memory instead of the difficulty that comes from replacing entities? For example, you can compare factual and fictitious questions when the numerical difficulty is the same.

Can you check that deos the recovery depends on that which entities remain factual?

In 4.3 can you compare the same checkpoints of the model with and without CoT prompting to see that does the factual-fictitious gap changes in the same way?

4 comments
AuthorsOpenReview ↗

Additional Controls on Numerical Difficulty, Entity Replacement, and CoT

We thank you for your careful reading and constructive feedback. We particularly appreciate your recognition of the novelty and strength of MemoReason's paired design: "Their design is solid [...] and it is novel". We also appreciate your assessment that "the way that they consider four different question types and three different answer types is solid". Your suggestions strengthened our controls and clarified our interpretation.

Weakness 1: "[...] do the models perform lower because that the info is not familiar to them, or because that the entities that have been replaced make the questions more difficult. [...]"

→\rightarrow Q4: "[...] you can compare factual and fictitious questions when the numerical difficulty is the same."

Thank you for raising this concern. Each substitution follows template rules on value ranges, ordering, offsets and arithmetic relations (Figure 1, rules panel), and fictitious performance is averaged over K=10 variants to reduce sensitivity to individual instantiations. To test this directly, MemoReason has a names-only configuration that keeps all numerical/temporal entities unchanged. We have added the results in Table 8 of Appendix G.1.2 (in blue): five of eight models still show a significant drop, so the effect appears without any change to the numbers and cannot be attributed to instantiation difficulty.

We also agree that extraction should be distinguished from reasoning. Table 2 reports extractive questions separately from the Reason. aggregate. Their drops show sensitivity to replacement even without numerical computation. We will clarify this distinction.

Weakness 2: "[...] The thing that they mention about the reason for recovery in figure 2 is not convincing [...] the result can be dependent on that what entities are replacing. [...]"

→\rightarrow Q5: "Can you check that deos the recovery depends on that which entities remain factual?"

Partial replacement never produces incoherent documents: entities are replaced as units, linked attributes move together, and every partially replaced document is verified against all template rules (Section 4.2). We vary which anchors remain factual, and Figure 2 averages over random selections. Following your request, we quantified it: at 50% replacement, 56.5% of the main entities retain a real name, falling to 22.2% at 80% and 10.7% at 90%, whereas the corresponding fractions of unchanged associated information are 51.6%, 23.6%, and 13.9%. Fewer factual anchors remain as the curves recover, consistent with reduced knowledge conflict (Longpre et al., 2021), though this is not causal evidence. We will include these results in Section 4.2.

Weakness 3: "[...] these two are different post-trained checkpoints [...]"

→\rightarrow Q6: "[...] can you compare the same checkpoints of the model with and without CoT prompting [...]?"

As stated in Section 4.3, that experiment tests the effect of CoT post-training, not CoT alone. For the latter, we ran your suggested control on Gemma, which supports a CoT on/off switch within a single checkpoint:

MeasureWithout CoTWith CoTChange, pp [95% CI]
Factual accuracy (%)84.3389.00+4.67 [2.44,6.89]
Fictitious accuracy (%)82.7885.89+3.11 [1.27, 4.94]
Factual–fictitious drop (pp)1.563.11+1.56 [−0.31, 3.44]

CoT significantly improves accuracy in both settings, while the drop persists and, if anything, widens (1.56 → 3.11 pp; the change itself is not significant). Within a single checkpoint, this reproduces the pattern of the post-training comparison: CoT raises accuracy, not robustness to entity familiarity.

Q1: "In line 333 [...] Can you report the median instead?"

Thank you; we will report the median: a 4.6-percentage-point decrease across the nine models on temporal questions with variant answers. We will clarify that Table 2's Δ is the mean paired difference (pp) defined in Section 3 and cross-reference l.303 to l.271.

Q2: "why are you not passing everything to the judge [...] Having a uniform evaluation is better"

Exact match is a conservative first filter, as it compares raw answers with the ground truth; only EM-negative predictions go to the judge. To validate it, we manually checked 300 randomly selected exact-match positives and found no false positives in either setting.

Q3: "In 2.2, can you report the inter-annotator agreement?"

The annotation process comprised two independent reviews followed by consensus sessions in which annotators resolved their disagreements. Initial inter-annotator agreement was 76.22% and final agreement reached 100%, reflecting the substantial reconciliation work undertaken during the consensus sessions.


Note: our reply to Reviewer nmc2 also reports new analyses of readability, entity token length and log-likelihood.


Thank you again for your constructive feedback. We hope these additions support a higher overall assessment.

AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer 9dwv,

As the discussion period draws to a close, we would like to kindly follow up on our response of 11 November.

We really appreciate your thoughtful suggestions, which helped us strengthen the paper. Following your feedback, we ran additional experiments keeping numbers and dates unchanged (which we added to the paper -- Table 8 in blue) and comparing the same model checkpoint with and without CoT. We also provided further analyses of entity replacement, manually checked 300 answers accepted by exact match, reported inter-annotator agreement, and responded to each of your questions.

Could you please let us know whether these additions and clarifications address your main concerns, or whether any points remain unresolved? We would be happy to provide any further clarification.

Thank you again for your time and consideration.

Thank you for the detailed response and the additional experiments and analyses. The names-only control and the comparison with and without CoT within the same checkpoint address my main concerns. I also appreciate the clarifications regarding entity replacement, evaluation, and inter-annotator agreement. I will increase my soundness score and maintain my positive overall assessment.

AuthorsOpenReview ↗

Big Thanks! :)

We sincerely thank Reviewer 9dwv for the positive follow-up, for acknowledging our additional experiments and clarifications, and for recognizing that they address the main concerns raised in the review. We greatly appreciate the increased soundness score and the reviewer's careful engagement with our rebuttal. Your constructive suggestions have helped us strengthen the paper, and we will ensure that these improvements are clearly reflected in the revised manuscript.

Thank you again for your time and valuable feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.