- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies historical financial recall across 60 LM configurations and 145 instruments. It combines direct price queries, dense daily probes, reverse instrument identification and corpus analysis, distinguishing fine-grained recall from coarse temporal estimates. The supplementary experiments examine robustness and how recalled information is used in historical prediction and simple decision tasks.
Critical strengths
- The daily-detail results are the strongest contribution. Figure 5's residual correlations, together with the daily-return analysis in F.12 and the large-move tests in F.13, provide convincing evidence that some models recover detail beyond a smooth price trend. The distinction between price-level accuracy and temporal detail is useful.
- The experimental coverage is substantial. Repeated sampling and paraphrases in F.3 provide useful robustness checks, while F.17 connects recall to historical prediction by varying the information supplied. The authors also report mixed results and acknowledge limitations, which makes the overall evidence more credible.
Critical weaknesses
- My main concern is the incremental contribution. Lopez-Lira et al. (2025) already examine financial recall, cutoff effects and recovery of identifiers from masked text; Kandpal et al. (2023) connect factual recall to the frequency of relevant training information. The 60-configuration comparison, dense daily analysis and reverse lookup from numerical traces are meaningful extensions. Sections 5, F.11 and F.15 also offer partial explanations involving corpus coverage, document type and volatility. These analyses remain correlational, however, and the additional insight is mainly a finer characterization of variation across models, instruments and retrieval cues. I find this useful, but the advance over prior work is still somewhat modest.
- Table 3 reports 78.1% for Sol and 68.8% for Opus on normalized, dated k=10 traces. Without dates, the corresponding results are 18.8% and 16.7%, above the 6.25% chance level, so there is evidence of identifying signal in the prices. What remains unclear is how much prices add once dates and candidates are known. Dates can already narrow the candidates through instrument lifetimes, and no matched dates-only comparison is reported. This limits what the normalized, dated results establish about detailed price knowledge.
Questions for authors
Could you report the normalized, dated k=1 identification results from the existing experimental grid? Also, for the exact k=10 questions in Table 3, how well do the models perform with all ten dates and the same candidate list, but no prices? Since windows are sampled separately across k, the k=1 result cannot serve as that matched control. This comparison would clarify how much the normalized price trace contributes.