acceptodds
Under review as a conference paper at ICLR 2027

Distant Memories: How do LLMs remember financial data?

Abstract

The nature of information storage and access in large language models is far from fully understood. We focus on knowledge of financial information, and to this end investigate the depth and nature of LLMs' knowledge of prices of instruments such as stocks, currencies and indices. We also investigate how this varies with the size and type of model, and whether it is affected by how the information is recalled. To yield a baseline, we investigate the retrieval of non-existent instrument prices, including those that are similar to existing instruments. We show that LLMs store data in different ways depending on the frequency of it appearing in the training data; we also show that the retrieval process is a two-way process where certain information is more helpful in retrieval than others.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, memorization, retrieval, finance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6rvsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies historical financial recall across 60 LM configurations and 145 instruments. It combines direct price queries, dense daily probes, reverse instrument identification and corpus analysis, distinguishing fine-grained recall from coarse temporal estimates. The supplementary experiments examine robustness and how recalled information is used in historical prediction and simple decision tasks.

Critical strengths
  1. The daily-detail results are the strongest contribution. Figure 5's residual correlations, together with the daily-return analysis in F.12 and the large-move tests in F.13, provide convincing evidence that some models recover detail beyond a smooth price trend. The distinction between price-level accuracy and temporal detail is useful.
  2. The experimental coverage is substantial. Repeated sampling and paraphrases in F.3 provide useful robustness checks, while F.17 connects recall to historical prediction by varying the information supplied. The authors also report mixed results and acknowledge limitations, which makes the overall evidence more credible.
Critical weaknesses
  1. My main concern is the incremental contribution. Lopez-Lira et al. (2025) already examine financial recall, cutoff effects and recovery of identifiers from masked text; Kandpal et al. (2023) connect factual recall to the frequency of relevant training information. The 60-configuration comparison, dense daily analysis and reverse lookup from numerical traces are meaningful extensions. Sections 5, F.11 and F.15 also offer partial explanations involving corpus coverage, document type and volatility. These analyses remain correlational, however, and the additional insight is mainly a finer characterization of variation across models, instruments and retrieval cues. I find this useful, but the advance over prior work is still somewhat modest.
  2. Table 3 reports 78.1% for Sol and 68.8% for Opus on normalized, dated k=10 traces. Without dates, the corresponding results are 18.8% and 16.7%, above the 6.25% chance level, so there is evidence of identifying signal in the prices. What remains unclear is how much prices add once dates and candidates are known. Dates can already narrow the candidates through instrument lifetimes, and no matched dates-only comparison is reported. This limits what the normalized, dated results establish about detailed price knowledge.
Questions for authors

Could you report the normalized, dated k=1 identification results from the existing experimental grid? Also, for the exact k=10 questions in Table 3, how well do the models perform with all ten dates and the same candidate list, but no prices? Since windows are sampled separately across k, the k=1 result cannot serve as that matched control. This comparison would clarify how much the normalized price trace contributes.

Reviewer_jKkoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how LLMs recall historical financial data from parametric knowledge without external tools. It evaluates 60 LLM configurations on 8,096 historical price questions across 145 financial instruments, examining how recall accuracy varies across models, instruments, and time periods. Through dense daily-price evaluations, the authors identify three recall regimes: Lookup Table, Anchor Points, and Coarse Level Prior. The paper also investigates reverse lookup, showing that temporal information substantially improves instrument identification, and analyzes the relationship between financial text exposure and recall accuracy.

Critical strengths

The paper provides an extensive empirical evaluation across a diverse set of models and financial instruments. The daily-price analysis offers an interesting perspective beyond aggregate recall accuracy, while the reverse-lookup and corpus analyses provide complementary insights into LLMs' historical financial knowledge.

Critical weaknesses
  1. Insufficient motivation and unclear practical significance

My primary concern is that the paper does not sufficiently establish why fine-grained memorization of historical financial prices is an important research problem. While the authors argue that parametric financial knowledge can support financial reasoning, this does not explain why accurately recalling daily price fluctuations matters in realistic applications, particularly when historical prices can be readily retrieved through external tools. More importantly, the paper does not adequately investigate whether such memorization improves financial decision-making or instead introduces risks such as look-ahead bias in historical evaluations. Although Appendix F.17 includes downstream experiments, they do not directly establish the practical consequences of the central memory behaviors identified in the paper. Consequently, the work provides an extensive characterization of an interesting phenomenon without sufficiently demonstrating why it matters for how financial LLMs should be evaluated, trusted, or used.

  1. Limited explanatory depth and generalizability of the proposed recall regimes

I find that the analysis of three proposed recall regimes remains largely descriptive. The experiments demonstrate different levels of historical price recall fidelity but provide limited insight into why these regimes emerge or what factors determine them. The daily-level evaluation covers only 16 instruments, each with a single three-year window, making it difficult to disentangle instrument characteristics from historical period effects or establish whether the regimes generalize. Similarly, the reverse-lookup experiments do not fully isolate whether the benefits of providing dates arise from accessing memorized price trajectories or exploiting temporal shortcuts, such as instrument lifetimes and historical events. Therefore, although the paper identifies distinct recall patterns, it offers limited explanation of the conditions underlying these patterns and their robustness across different settings.

  1. Insufficient coverage and positioning of closely related literature

I find the related work discussion incomplete, particularly given the substantial existing literature on factual knowledge biases and financial memorization in LLMs. Several closely related studies are missing [1,2,3,4]. These omissions matter beyond citation completeness. Several findings emphasized in this paper, including stronger recall for prominent financial instruments, temporal variation in historical knowledge, and the association between information exposure and recall accuracy, substantially overlap with established observations. Without adequately discussing these studies, the paper does not clearly distinguish which findings represent new insights and which primarily extend or reproduce known phenomena in a different experimental setting. I think this weakens the positioning of the paper's novelty, particularly since its most distinctive contributions appear to lie in characterizing daily-level recall fidelity and reverse retrieval rather than in establishing these broader knowledge biases.

[1] Shah et al. (2025). Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge. COLM 2025.

[2] Didisheim et al. (2025). AI's Predictable Memory in Financial Analysis. Economics Letters.

[3] Mallen et al. (2023). When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.

[4] Sun et al. (2024). Head-to-Tail: How Knowledgeable Are Large Language Models (LLMs)? NAACL 2024.

Questions for authors

See weakness above.

Reviewer_jdPsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper assesses whether LLMs have an intuition about historical numbers, using financial data as a lens. It examines (1) how models remember financial instrument prices, i.e., at a fine-grained (daily) or coarse-grained (monthly) level, (2) whether a model can identify an instrument from its prices, and (3) whether the fame of an instrument affects the model’s accuracy on its prices.

Critical strengths

S1. Even in the era of tool calling, a statistical intuition about numbers is important for models, particularly when tool results may be erroneous or unreliable. In this sense, assessing such intuition of language models in a systematic manner is an important and timely direction.

S2. The high-level experimental design is sensible. Financial instrument prices are a kind of quantity that people (and LLMs) may have a rough feeling for without remembering the actual individual numbers. Using them as a lens of analysis is a good choice.

S3. The volume, coverage, and statistical treatment of the experiments are sufficient.

Critical weaknesses

W1. The paper does not form a story; it lists observations. The introduction raises interesting research questions: does the model understand or memorize the numbers, and how does it store such information in its parameters? However, the following sections are not designed to answer these questions. The three regimes in Section 3 are an example. They are not derived from a hypothesis or a line of reasoning, but are named after patterns observed in the results. They are also not precisely defined: what does “closely” reproduced mean, and what is a “rough” price level? Similarly, the questions of Sections 4 and 5 are not motivated in the introduction; they are only listed in its last paragraph, without a rationale. Why should the question be reversed? Why should the correlation between the amount of financial text and recall error be measured? As a reader, I can sense that these questions are related to the main research questions, but the link between them is missing.

W2. Section 5 does not answer its own question. According to its first paragraph, the question is about the type of information (Quote: “But what kind of information is it? Do the prices come from tables, newspaper articles, reports, or something else?”). However, the experiments that follow are all about the amount of information, which does not answer this question. The section title “Data Provenance” is therefore also inadequate.

W3. Some claims and design choices in Section 5 are hard to follow. Given that OLMo 3 is trained on Dolma 3, why is the analysis not conducted between OLMo 3 and Dolma 3? For the language coverage claim, the Toyota analysis is hard to digest: the framing sentence does not explain why Toyota is chosen, or what the OLMo/Apertus comparison is meant to establish. In addition, 37 of the 95 equities are excluded because their phrase-matching counts are zero. This seems unrealistic, and may indicate a problem in the matching process or in the instrument selection.

W4. The reason for choosing 16 candidates in Section 4 is not explained. Section 3 reports extensive results on 145 instruments, whereas Section 4 uses only 16. If these 16 candidates differ substantially in price level, the task becomes much easier, which means the selection would then have a large effect on the results.

W5. The presentation needs improvement. Each sentence is usually fine on its own, but the connections between sentences are often missing: why an experiment is necessary and how it relates to the main research question, what each number means, and what high-level insight follows from the results are left for the reader to reconstruct. The following points should also be addressed.

  • The metrics are mixed. Table 2 reports median |ln| error, the “within 5%” column is a percentage, and the main text uses percentage error. In addition, the log error understates the severity of large errors.
  • The abstract lists the experiments rather than conveying a high-level insight.
  • The main text of Section 3 describes four curves (model answers, ground truth, interpolation of model answers, interpolation of ground truth), but Figure 5 shows only three.
Questions for authors

Q1. Please clarify the overall framing of the paper: how each section is meant to answer the research questions raised in the introduction, and what the high-level conclusion is. Please also improve the connection between sentences and paragraphs throughout. See W1 and W5.

Q2. Please explain the rationale behind the design choices in Section 5, in particular why the analysis measures the amount of coverage rather than the type of source, why Toyota is used for the language coverage claim, and why 37 equities have zero counts. See W2 and W3.

Q3. Please explain why Section 4 uses 16 candidates instead of the 145 instruments of Section 3, and how the price levels of the candidates affect the difficulty of the task. See W4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.