acceptodds
Under review as a conference paper at ICLR 2027

Detecting Benchmark Memorization in LLMs on Numerical Optimization Tasks

Abstract

Large language models (LLMs) give accurate answers on classic test functions. Shown samples of an unnamed function, they often return its textbook optimum to four decimals, which raises the question of whether they are inferring the answer from the data or recalling the benchmark. We propose a memorization-controlled protocol for numerical optimization on continuous test functions, with integration and multi-fidelity prediction as control tasks. Our protocol includes: (i) a recall signature, defined as the near-exact answer, which no data-driven surrogate attains on any optimization or multi-fidelity cell; (ii) a procedural generator that moves each benchmark optimization function away from its published form by adding a random smooth field, whose amplitude in units of the function's own variability sets the distance, yielding a measured recall dose-response; and (iii) a surrogate floor that distinguishes anchoring on the benchmark answer from defensible shrinkage toward it. Across sessions with seven LLMs, models returned near-exact answers on benchmark optimization problems in up to 88% of sessions, even when given fresh samples and opaque identifiers. In sessions on novel functions, they never did. Models returned the benchmark answer less often as functions moved farther from the benchmark, and how quickly this happened varied by model. GPT and Claude models returned the benchmark answer in 23–78% of sessions even when it was strictly worse than an estimate fitted to the samples shown, and this held across every surrogate floor we tested. This suggests that models recognize a benchmark when deviations fall within 1–5% of the spread in the samples. Newer Claude and GPT models relied on benchmark answers more often than their predecessors, and for Claude, higher reasoning effort increased it as well; code execution did not eliminate recall. Whether recall occurred depended on the particular samples shown as well as on the function. Therefore, accuracy on classic test functions alone does not establish numerical reasoning; the protocol offers a reusable way to evaluate numerical performance while controlling for memorization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs for optimization, numerical integration, multi-fidelity, evaluation protocols, benchmark memorization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gKVeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether LLMs solve familiar numerical optimisation problems by recalling published answers rather than reasoning from the samples given. They test released benchmarks, fresh samples, and perturbed functions. They track how often models return the canonical optimum as perturbation grows, and compare against surrogate fits. Extra experiments cover reasoning effort, model generations, sample draws, and code execution.

I find the central observation convincing: models often return familiar benchmark answers, and sometimes continue doing so after the underlying function has changed. This is a useful warning for evaluating numerical capability. My concerns lie in the stronger interpretation of this behaviour and in whether the scoring establishes that the particular answer returned was worse than the surrogate estimate.

Critical strengths

The question is worthwhile and the effort is substantial. Fresh samples, opaque identifiers, perturbation seeds, and the librarian reference give several complementary probes. The continuous perturbation ladder is a useful addition over earlier transformation-based work.

I appreciate the recognition that authors note a small perturbation may make the canonical answer still rational. Comparing against a fitted surrogate is a sensible way to separate useful priors from harmful anchoring. Testing several surrogate floors helps. The authors are open about limitations: tool-arm failures, surrogate-floor sensitivity, low power of the matched comparison.

The fresh-draw experiment is also valuable. Recall rates can vary considerably even when the function and sample budget remain fixed. That finding matters for anyone constructing small numerical benchmarks.

Critical weaknesses

"Recall" is defined to include recognising a closed form and computing its optimum. But these are different operations. Retrieving a stored constant, identifying a function family, and computing a consequence from that family are distinct mechanisms. The selected surrogates fail to hit the near-exact threshold. That separates the result from those surrogates. It does not show no fitting method could recover the answer. Identifiability depends on the hypothesis class. The evidence supports strong dependence on familiar benchmark structure. Claims about the internal mechanism, or about the absence of numerical reasoning, need more qualification. A control using random input translations or rescalings could help distinguish reuse of an answer constant from recognition followed by calculation.

Section 3 counts canonical emission if either location or value matches. Appendix A.5 labels a cell pathological if either canonical location or value loses to the GP floor. These two disjunctions don't align. A model could match the canonical value on a cell labelled pathological solely because the canonical location is poor. Counting that session does not show the reproduced component was worse. The components are being conflated. Also: the floor compares canonical minimum-value error with GP optimisation regret. Those are different quantities. Regret at a point is not the same as predicted-minimum error.

A large number of sessions describe behaviour conditional on the tested prompts. But generalisation rests on a small set of base functions, perturbation fields, and sample draws. The redraw results make this clear: emission rates go from 0% to 100%, and surrogate-floor classification flips with the draw. The authors note this, but it directly affects the uncertainty on the main claims. Also, perturbation seeds reused across ladder rungs introduce shared structure --- a violation of the independence assumption behind naive error bars. The authors should report uncertainty that clusters by perturbation field and base function, not per-cell. Replicating sample draws in the central harmful-anchoring comparison would be especially useful.

Questions for authors

Do the harmful-emission rates remain similar when location and value are scored separately against matching losses?

What happens to the surrogate near-exact rates after continuous refinement?

How are multiple global minima handled in the location metric?

Do the main conclusions survive clustering by perturbation field and replication over fresh sample draws?

Can a simple transformed-function control separate remembered constants from recognition followed by calculation?

2 comments

I see that the authors have basically addressed all of my comments. I've raised my score to weak accept. The current experiments select some harmful and some favrouable draws per function from a number of candidates. To answer "across randomly sampled problems, how often does a model return a harmful canonical location?" The way they are selected may need some careful consideration in future work.

Thank you for the reassessment. We appreciate it and agree with your remaining point: the 40% and 33% are rates on selected harmful draws, so they do not say how often a model returns a harmful location on a randomly drawn problem. The revision states this as a limitation, and measuring the unconditional rate on unselected draws is future work.

Reviewer_78QdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evaluation framework to investigate whether LLMs, when addressing numerical optimization tasks, solve them based on the provided samples or simply retrieve answers associated with known benchmark functions. The framework incorporates benchmark and reconstructed instances, controlled perturbations, comparisons with surrogate models, and a librarian baseline.

Critical strengths
  1. The paper addresses a clear evaluation problem: strong scores on familiar benchmarks may not reflect reasoning from the provided data.
  2. The paper combines several controls: regenerated samples, opaque identifiers, novel functions, reference models, and a tool-use arm.
Critical weaknesses
  1. My main concern is that the evidence supports benchmark-answer emission more directly than it supports the stronger claims that the models disregard the data or that the protocol identifies memorization. The author is suggested to be careful about this claim.
  2. In my opinion, surrogates failing to meet a threshold on these cells does not establish that data-driven inference cannot meet it; likewise, a librarian designed around six known functions is not a general test of recognition.
  3. The paper notes that panels differ in more than model version, and the effort comparison is within one model. These results are suggestive, but they do not show that newer models or additional reasoning cause more recall in general.
  4. IMHO, the major contribution is an evaluation protocol plus evidence of benchmark-answer anchoring on selected low-dimensional functions. Claims about general numerical capability, memorization mechanism, and trends across generations need substantially more qualification.
  5. I am confused by the contribution of the paper. I would regard this paper as a technical report rather than a research paper. I hope the author can defend this point.
Questions for authors
  1. Do the results show that models disregard the data, or only that they sometimes return familiar benchmark answers? Could the claims about detecting memorization be stated more cautiously?
  2. Do the surrogate results show that data-driven methods cannot reach the near-exact threshold, or only that the tested surrogates did not? How broadly can a librarian built around six functions test recognition?
  3. Since the model panels differ in more than version and the effort comparison covers only one model, can the paper support general claims that newer models or more reasoning increase recall?
  4. Can the authors clarify which conclusions extend beyond the selected low-dimensional functions, and qualify claims about numerical ability, memorization mechanisms, and model-generation trends?
  5. What is the paper’s main research contribution beyond documenting benchmark-answer anchoring? Why should it be considered a research contribution rather than a technical report?
Reviewer_8f7oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines what happens when an LLM must optimize, integrate, or make multi-fidelity predictions from only a few samples of an anonymized function, and it designs a protocol to keep memorization in check. The protocol has three parts: (1) Recall signature: A "near-exact" answer: one that no data-driven surrogate ever reaches, while a "librarian" holding the closed forms always does. (2) Perturbation ladder: measuring "canonical emission" as a dose-response in ε. (3) Surrogate floor: A GP fit to the shown samples; a perturbed cell is flagged "pathological" when the canonical answer is worse than the fit.

Critical strengths
  • The question matters and is timely: the LLM-as-optimizer/regressor literature is evaluated on exactly these test functions.
  • The protocol is carefully thought through: opaque IDs, a regenerated-vs-released split, a librarian reference, a distance axis with computable regret, multiple floors, pre-registered tool-arm predictions, cell-cluster bootstrap, and an answer-collapse audit, all of which go well beyond a standard contamination study.
  • The authors are unusually frank about the study's limitations (draw variance, the matched-strata power limit, panel confounds).
Critical weaknesses

1. The recall signature is weak on functions whose optima are "round". Matching x* and f* within 1e-3 is a strong recall signal for Branin and Hartmann, whose optima are irrational constants, but a much weaker one for:

  • Ackley: x* at the domain center (0.5, 0.5), f* = 0.
  • Goldstein-Price: x* = (0, −1) → (0.5, 0.25) in the unit square, f* = 3.
  • Rosenbrock: f* = 0.

On these functions, an agent carrying a generic "simplicity prior" can arrive at a near-exact answer without recognizing the benchmark at all: it sees small values near the center or near a lattice point, snaps to round coordinates, and predicts an integer minimum. The data-driven surrogates never do this, since they are grid minimizers with no such prior, so the 0/1356 does not rule it out.

This directly affects the anchoring result. The paper itself notes that pathological cells concentrate on Goldstein-Price and Rosenbrock (13 of 16), and that the effort effect "falls mostly on Goldstein-Price, whose canonical target needs no numerical precision." The f*-channel criterion is especially exposed

2. The "pathological" label is a noisy, post hoc, unobservable quantity, so "models disregard the data" is not established. A cell is called pathological when the realized error of one GP fit on ten points (median argmin error 0.12) exceeds the canonical answer's drift. The model cannot see this label, and the label is highly variable: it flips across sample draws in two of four cells (Table 6B), and its membership changes across fitters (24/16/15/17 cells). The paper's own matched-stratum analysis (Fig. 3b) finds no sensitivity of emission to the verdict. So the evidence is consistent with a model following a fixed, ε-dependent policy, which is exactly what one expects if the verdict is mostly noise relative to what the model can observe..

Questions for authors
  1. With round-number functions excluded, what are the near-exact rates on regenerated cells and the pathological emission rates?
  2. On MF, GPT-5.6 is 81% near-exact, which requires evaluating the high-fidelity closed form at new points. On INT it is only 5%. Does not this suggest that MF retrieves the closed form (not just stored constants)? Could the INT/MF contrast separate "stored optimum constants" from "recognized closed form", which the paper currently lumps together as recall?
  3. Integration is presented as a control. But no canonical integral values are famous, so the near-exact signature cannot fire by recall there. In what sense is INT a negative control for recall, rather than a task where recall is simply unavailable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.