- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether LLMs can infer and reuse an estimator from compressed results of previous analyses. The authors use causal-effect estimation and meta-analysis tasks with deterministic scoring. They compare incorrect archived estimates against several controls, including correct estimates, and test how changes to the archived information affect downstream estimator choice. The results show that two Qwen models can reuse an incorrect estimator when the archived information makes it identifiable. This effect is reduced or absent when the identifying information is removed, and it varies across models and prompting conditions.
Critical strengths
-
Careful experimental design and validation. The paper uses well-designed controls and interventions to distinguish alternative explanations for estimator reuse. In particular, the correct-answer control and the non-identifying statistic intervention strengthen the evidence for the proposed mechanism. The deterministic scoring, pre-registered analyses, and transparent reporting of negative results further improve the credibility of the findings.
-
Focused and coherent argument. The paper maintains a clear focus on its central research question. The experiments are systematically organized to establish the phenomenon, examine its possible explanations, test the proposed mechanism, and explore its limitations. This creates a coherent and comprehensive body of evidence, with each experiment serving a clear purpose in the overall argument.
Critical weaknesses
-
Novelty and scientific significance could be better established. While the paper introduces a careful experimental framework, it remains unclear how much new understanding of LLM behavior the findings provide beyond existing research on answer conditioning and context-dependent algorithm selection. In particular, the paper could better explain what broader insight the proposed recoverability mechanism offers. Its practical implications for scientific AI workflows also remain largely untested.
-
The proposed mechanism is not fully distinguished from alternative explanations. The experiments show that numerical relationships in archived results influence downstream estimator choice. However, it remains unclear whether the models actually reconstruct an estimator or simply imitate numerical patterns and output templates. As also acknowledged by the authors, the current controls do not fully separate estimator identification from numerical agreement. This distinction is important because estimator recoverability is the central explanation proposed by the paper.
Questions for authors
-
Would the same estimator reuse effect occur if the archived answer matched a displayed statistic that was not an estimator output?
-
Can the five-control attribution result be replicated with a pre-registered memo-stage outcome, without relying on the decoding-rule probe?
-
Does the effect persist under weaker transfer instructions (P0–P2) when evaluated using the same memo-stage protocol as P3?
-
Can the effect be demonstrated in another model family or estimation task where the estimator is recoverable, the positive control is responsive, and the baseline leaves room for an effect?