- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MemoReason, a paired question-answering benchmark constructed by replacing entities in factual Wikipedia-based texts with fictitious entities governed by consistency rules, so that task structure is preserved while familiarity with the content varies. The authors design variant-answer, invariant-answer, and refusal questions, together with extractive, arithmetic, temporal, and inference tasks, to compare model performance on familiar factual texts and their fictitious counterparts. Experiments show that reasoning performance generally declines on fictitious texts, but errors rarely directly reproduce the corresponding factual answers, leading the authors to argue that a strong shortcut hypothesis based on skipping reasoning and recalling memorized answers is not the dominant failure mode. The paper further examines the effects of partial entity replacement, chain-of-thought, and reasoning effort on the performance gap, and releases the dataset, code, and annotation workflow.
Critical strengths
- The paired factual/fictitious design provides clear experimental control, helping to distinguish changes in content familiarity from changes in task structure.
- The benchmark construction incorporates human review, entity-type annotation, consistency-rule constraints, and public resources, making the work relatively transparent and reusable.
- The paper goes beyond reporting an overall gap by analyzing it across question types, replacement proportions, reasoning settings, and direct answer reproduction.
Critical weaknesses
- Although fictitious entities are type-matched, they may still introduce hidden changes in linguistic distribution, readability, plausibility, or question difficulty, and these confounding factors are not sufficiently ruled out.
- The interpretation of “memorization bias” remains mainly behavioral, without more direct mechanistic evidence, which makes the causal conclusions difficult to support strongly.
- The tasks are focused on Wikipedia-style factual texts and a limited set of question types and models, so the generalization to broader contextual reasoning capabilities remains insufficient.
Questions for authors
- How do you rule out the possibility that the performance change is driven by differences in the linguistic distribution, readability, or plausibility of the fictitious entities themselves, rather than by familiarity with parametric memory?
- Beyond observing final output behavior, have you considered using internal representations or stronger interventional experiments to verify the causal mechanism behind the proposed memorization bias?
- Since the benchmark mainly relies on Wikipedia-style factual texts and a limited range of question types and models, on what basis do you believe the conclusions remain valid in more open-ended contextual reasoning settings?
4 comments
Rebuttal Follow-up
Dear Reviewer nmc2,
As the discussion period draws to a close, we would like to kindly follow up on our response of 11 November.
We really appreciate your thoughtful feedback, which helped us strengthen the evaluation and clarify our conclusions. Following your suggestions, we provided additional experiments of readability and linguistic differences between the factual and fictitious texts, as well as a control experiment for task difficulty (which we added to the paper -- Table 8 in blue). We also clarified what our results do and do not establish about the causes of the performance gap and whether the findings extend to other reasoning tasks.
Could you please let us know whether these additions and clarifications address your main concerns, or whether any points remain unresolved? We would be happy to provide any further clarification.
Thank you again for your time and consideration.
Thank you for the author's serious response. The author's response has largely resolved my concerns about reliability and generalization, so I have raised my score to 3
Big Thanks! :)
We sincerely thank Reviewer nmc2 for the positive follow-up and for recognizing that our additional experiments and clarifications have largely addressed the concerns about reliability and generalization. We greatly appreciate the increased score and the reviewer's thoughtful engagement with our rebuttal. Your constructive feedback has helped us strengthen the evaluation and clarify the scope of our conclusions, and we will ensure that these improvements are clearly reflected in the revised manuscript.
Thank you again for your time and valuable feedback!
Additional Experiments on Readability, Task Difficulty, and Familiarity, and Clarification of our Conclusions
We thank you for your constructive assessment and for recognizing the design's "clear experimental control," the benchmark as "relatively transparent and reusable," and the analysis that "goes beyond reporting an overall gap." We address the three concerns below.
Thank you for identifying these possible confounds. Our construction limits these effects through type-compatible substitutions, consistency rules, coherent entity attributes, and double blind human validation. To further investigate these issues, your feedback motivated the additional checks below.
1. Readability and plausibility:
Our generation prompt explicitly requests readable and plausible entities (l.1705–1706):
Following your suggestion, Flesch-Kincaid grades are 14.25 (factual) and 14.47 (fictitious): +0.22 [95% CI: −0.014, 0.43], a small, nonsignificant difference. We will add this analysis of human readability to the camera-ready version.
2. Task difficulty:
Replacements are sampled under template rules that constrain value ranges and preserve order, offsets, and arithmetic relations, so instantiation difficulty varies symmetrically around the factual values rather than in a principled direction, and every reported number averages ten variants. To address this concern (also raised by Reviewer 9dwv),
MemoReasonincludes a configuration where only named entities are replaced and all numerical/temporal entities are kept unchanged, leaving the numerical difficulty unchanged. Table 8 (Appendix G.1.2, in blue in the revision) shows significant drops for five of eight models: therefore, harder numerical calculations cannot be the only explanation.3. Linguistic distribution:
We agree with the reviewer that there is a shift in the linguistic distribution, and this is exactly what MemoReason intends to test: what happens to LLMs’ performance when familiar content is replaced with unfamiliar content while the task structure is preserved? Our hypothesis is that reduced familiarity may contribute to the observed performance drops.
Using Qwen3.5-27B, we computed mean document log-likelihood from the probabilities assigned to successive tokens given their preceding text. These probabilities reflect patterns learned during training, making log-likelihood a model-relative indicator of statistical familiarity. We also measured the mean token length of the annotated entity surface forms:
Fictitious documents have significantly lower log-likelihood, consistent with reduced statistical familiarity under this model, and their entities are longer on average. These diagnostics quantify the distribution shift without establishing the cause of the performance gap. However, changes in log-likelihood show a weak, nonsignificant correlation with performance drops across documents. Larger measured shifts do not consistently correspond to larger performance losses.
We agree that our evidence is behavioral rather than mechanistic, consistent with the scope of this datasets and benchmarks paper. As noted in the conclusion (line 477): "Identifying the mechanisms underlying these performance differences remains an open question [...]". To support this research, we release our annotation interface, code, and all MemoReason configurations (109,200 examples; anonymous links in page 1’s footnote).
Thank you for raising this point. We agree that our results on Wikipedia-based QA do not establish generalization to open-ended reasoning, and we will explicitly mention this limitation in the paper. Our released annotation interface and guidelines make it easier to extend MemoReason to other text genres and tasks, and to test whether our findings hold in those settings.
Note: we have also conducted new experiments on chain of thought effect (see answer to reviewer 9dwv).
Thank you again for helping us strengthen the evaluation and clarify our conclusions. We hope these clarifications and new experiments will support a higher assessment.