acceptodds
Withdrawn from ICLR 2027

Estimator Recoverability, Not Numeric Carriage, Explains Estimator Reuse from Compressed Scientific Results

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific AI workflows increasingly pass results between models in compressed form: a reported estimate survives, while the analysis that produced it does not. When an upstream analysis used a flawed estimator and a downstream model later applies the same estimator to a new problem, it is tempting to conclude that the archived number carried the method. Yet recurrence alone cannot show this: the task, the reuse instruction, the presence of an artifact or the model's own prior can produce the same choice, and studies of answer cues lack the control that separates estimator-specific content from generic answer conditioning—a correct archived answer. We introduce matched-carrier attribution, which credits an archived wrong answer with carrying an estimator only if it induces that estimator on an independent new problem more often than every matched alternative, the correct answer included. We apply it to causal-effect estimation tasks in which a deterministic oracle identifies the estimator a model uses, and locate the mechanism with pre-registered interventions that edit one sentence of the artifact and leave the archived number unchanged. In Qwen3.5-27B, an archived wrong answer reliably leads the model to adopt the confounded estimator on new problems and exceeds all five controls. The effect disappears whenever the artifact no longer lets the estimator be identified, including when the quantity the number was computed from is replaced by a statistic of the same form that keeps the artifact complete, and this dependence replicates in a newer model generation, Qwen3.8-27B. Where the artifact never permits identification, as in meta-analysis, the number has no effect. Recoverability is necessary but not sufficient: when the model must end with a numeric estimate or declare the effect not identifiable, the effect shrinks in Qwen3.5-27B with the same dependence, and a second family, Mistral Small 3.2 24B, shows none. An archived number thus transmits a method not by itself but by letting the method be recovered from its context, which also means that a correct archive is not automatically safe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, scientific workflows, estimator reuse, estimator recoverability, identifiability, causal-effect estimation, matched controls, pre-registered interventions, provenance, compressed scientific results

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9af8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether LLMs can infer and reuse an estimator from compressed results of previous analyses. The authors use causal-effect estimation and meta-analysis tasks with deterministic scoring. They compare incorrect archived estimates against several controls, including correct estimates, and test how changes to the archived information affect downstream estimator choice. The results show that two Qwen models can reuse an incorrect estimator when the archived information makes it identifiable. This effect is reduced or absent when the identifying information is removed, and it varies across models and prompting conditions.

Critical strengths
  • Careful experimental design and validation. The paper uses well-designed controls and interventions to distinguish alternative explanations for estimator reuse. In particular, the correct-answer control and the non-identifying statistic intervention strengthen the evidence for the proposed mechanism. The deterministic scoring, pre-registered analyses, and transparent reporting of negative results further improve the credibility of the findings.

  • Focused and coherent argument. The paper maintains a clear focus on its central research question. The experiments are systematically organized to establish the phenomenon, examine its possible explanations, test the proposed mechanism, and explore its limitations. This creates a coherent and comprehensive body of evidence, with each experiment serving a clear purpose in the overall argument.

Critical weaknesses
  • Novelty and scientific significance could be better established. While the paper introduces a careful experimental framework, it remains unclear how much new understanding of LLM behavior the findings provide beyond existing research on answer conditioning and context-dependent algorithm selection. In particular, the paper could better explain what broader insight the proposed recoverability mechanism offers. Its practical implications for scientific AI workflows also remain largely untested.

  • The proposed mechanism is not fully distinguished from alternative explanations. The experiments show that numerical relationships in archived results influence downstream estimator choice. However, it remains unclear whether the models actually reconstruct an estimator or simply imitate numerical patterns and output templates. As also acknowledged by the authors, the current controls do not fully separate estimator identification from numerical agreement. This distinction is important because estimator recoverability is the central explanation proposed by the paper.

Questions for authors
  1. Would the same estimator reuse effect occur if the archived answer matched a displayed statistic that was not an estimator output?

  2. Can the five-control attribution result be replicated with a pre-registered memo-stage outcome, without relying on the decoding-rule probe?

  3. Does the effect persist under weaker transfer instructions (P0–P2) when evaluated using the same memo-stage protocol as P3?

  4. Can the effect be demonstrated in another model family or estimation task where the estimator is recoverable, the positive control is responsive, and the baseline leaves room for an effect?

Reviewer_J7DNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a specific question: when an upstream analysis archives only its final numeric estimate and discards the estimator that produced it, does a downstream model facing an independently generated new problem reuse that estimator? The authors observe that recurrence of the same estimator is consistent with two accounts — method induction (driven by the task, the transfer instruction, the presence of an artifact, and the model's own prior) and answer-specific reconstruction (the archived value itself carrying estimator-diagnostic information). The comparison required to separate the two — an artifact of the same form whose value is the correct answer — is absent from prior work on answer cues.

Critical strengths

Prior work compared against "no answer" or "a random number", so a positive result remained compatible with generic answer conditioning: a model that sees a plausible answer is more willing to reuse the naive method. The control added here matches the target artifact in format, length and plausibility and differs only in whether the value points at the correct rule, so it can separate "the number carries method information" from "answer conditioning". This is decisive for my overall assessment because the paper's stronger claim — that it is the wrong answer itself that acts — rests entirely on this contrast; and the result survives at the strictest comparison (0.988 versus 0.207), which shows that the attribution does not come from the easier "no artifact" comparison.

Critical weaknesses

The first issue is scope. All positive evidence comes from a single model–operator–prompt cell (Qwen3.5-27B / E1 / P3). I would be happy to raise my score if you could provide additional supporting evidence across more models. This cell was selected by the authors from 18 configurations on two eligibility conditions — a responsive positive control, and a no-artifact floor that leaves room for an effect to appear. The only other model family that satisfies the eligibility conditions shows no effect, and the paper reports that its cross-cell criterion (Criterion 2) fired.

Second, in the E1 construction "recoverable" is operationally close to "the archived value equals the slope displayed in the artifact". The authors state in Appendix B.4 that the positive result cannot be read as evidence of a hidden channel and should be understood as evidence that the model can do the arithmetic.

Together these limit the strength of the conclusion. The title and abstract offer a general explanation of estimator reuse, whereas the evidence supports a much narrower statement: that for a particular model family, operator and transfer prompt, an artifact containing a checkable field leads the model to rebuild the estimator. This matters for my overall assessment because readers may derive a general requirement for scientific handoffs (carry the method together with its assumptions, or carry no method-identifying context), while the central qualifier — that recoverability is necessary but not sufficient — is easily lost in citation.

Questions for authors

Q1. All sampled-probe evidence comes from P3, which explicitly asks the model to carry the upstream method forward; realistic handoffs rarely contain such an instruction. If the effect is zero at P0–P2, the paper is about instructed reuse rather than about compressed handoffs. Do the authors agree with this characterisation, and would they consider limiting the conclusion accordingly in the abstract?

Q2. The parent artifact is generated by deterministic code. A real upstream model's artifact would more plausibly name the method, add caveats, or state its assumptions. Would such an artifact make the estimator recoverable without the slope — thereby changing the mechanism — or would it make the artifact no longer recoverable? Have the authors run any pilot with model-generated parent artifacts?

Reviewer_7wbcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether a compressed scientific result can induce reuse of an incorrect estimator on an independent problem. It introduces a matched-carrier criterion requiring the wrong-answer artifact to exceed five controls, including a correct-answer artifact. The main assay distinguishes the causal effect from the confounded observational slope in a synthetic structural causal model. In Qwen3.5-27B, wrong-answer artifacts produce a large increase in wrong-estimator signatures. Removing the parent observational slope, or replacing it with a non-identifying statistic, substantially reduces the wrong-versus-correct archive contrast. The latter intervention is replicated in Qwen3.8-27B. An explicit final-answer format sharply reduces the effect in Qwen3.5, while an eligible Mistral model shows no detectable effect. The authors interpret these findings as evidence that estimator recoverability, rather than the archived number alone, drives reuse.

Critical strengths
  1. The matched controls, particularly the correct-answer archive, help disentangle estimator-specific transmission from generic reuse behavior.
  2. The paper presents informative negative results and demonstrates that an explicit final-answer instruction substantially attenuates the effect.
Critical weaknesses
  1. The practical motivation is not fully established by the current evaluation setting. The positive results rely on synthetic tasks and explicit method-reuse instructions, without modeling interactive clarification workflows. The paper does not demonstrate how consequential this failure would be in settings where assistants validate inherited methods or request missing information. In addition, the evaluation is limited to open-weight models.

  2. The explanatory and mitigation contributions remain limited. The input interventions identify influential contextual cues but do not fully disentangle estimator recovery from relational template copying, and no model-internal analysis is provided. Although A4 offers a useful observation about mitigation, the paper does not systematically evaluate a mitigation strategy or its associated trade-offs.

Questions for authors
  1. Does this failure remain consequential in realistic workflows where assistants can request clarification and are expected to validate inherited methods?

  2. How do the experiments distinguish genuine estimator recovery from copying relational patterns among fields in the parent template?

  3. Does the effect generalize to closed-source models? Does A4 reduce errors without inducing unnecessary abstention on tasks where the effect is identifiable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.